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PRIOR  KNOWLEDGE  ASSESSMENT  GUIDE 


EXECUTIVE  SUMMARY 


Research  Requirement: 

A  central  tenet  within  the  U.S.  Army  Training  and  Doctrine  Command’s  (TRADOC) 
Army  Learning  Concept  2015  (ALC)  and  Army  Learning  Model  (ALM)  is  the  need  to  transition 
to  learner-centric  methods  and  processes  that  develop  critical  competencies  through  rigorous, 
tailored,  outcome-oriented  learning  experiences.  Despite  this  goal,  the  ALC  and  ALM  do  not 
provide  guidance  on  how  to  effectively  identify  students  who  could  benefit  from  tailored  training 
experiences.  One  of  the  objectives  of  the  Army  Research  Institute’s  (ARI’s)  tailored  training 
program  is  to  determine  the  types  of  measures  that  predict  performance  in  Army  courses.  These 
measures  could  then  be  used  to  identify  students  likely  to  have  problems  in  the  course  or  those 
who  need  to  be  challenged,  so  instructors  could  address  the  needs  of  these  individuals  through 
tailored  training.  The  result  of  ARI’s  research  efforts  showed  that  relevant  prior  knowledge  was 
the  most  consistent  predictor,  a  result  consistent  with  the  academic  literature.  Since  neither  the 
ALC  nor  the  ALM  provided  guidance  to  instructors  on  how  to  identify  students  who  would 
benefit  from  tailored  training,  the  prior  knowledge  assessment  guide  presented  in  this  report  was 
developed  to  fill  that  training  gap.  It  was  based  heavily  on  the  lessons  learned  from  ARI’s 
tailored  training  research,  and  was  designed  to  assist  training  personnel  in  identifying  levels  of 
prior  knowledge  in  students  prior  to  instruction. 

Procedure: 

A  guide  was  developed  that  provides  step-by-step  instructions  for  developing  a  Prior 
Knowledge  Assessment.  The  guide  provides  background  infonnation  on  prior  knowledge 
assessments,  procedures  for  developing  a  prior  knowledge  assessment,  developing  questions, 
procedures  for  validating  prior  knowledge  assessments,  infonnation  on  using  and  revising 
assessments,  and  practical  exercises  to  assist  developers  in  understanding  major  concepts  and 
procedures  in  the  guide.  The  guide  was  distributed  to  three  selected  training  institutions  to 
obtain  feedback  on  the  content,  organization,  clarity,  ease-of-use,  readability,  and 
understandability  of  the  material  presented  in  the  guide. 

Findings: 

The  guide  was  demonstrated  to  contain  the  appropriate  information  developers  need  to 
create  Prior  Knowledge  Assessments.  Feedback  indicated  that  while  the  reviewers  thought  that 
the  guide  was  well  written  and  infonnative  for  creating  a  Prior  Knowledge  Assessment,  it  did 
include  some  concepts  that  more  junior  level  instructors  found  challenging  to  understand  and 
employ.  On  the  other  hand,  positive  feedback  was  provided  by  senior  level  personnel,  including 
senior  trainers,  training  managers,  course  developers  and  other  senior  training  personnel, 
regarding  the  guide’s  utility.  Thus,  it  appears  that  the  prior  knowledge  assessment  development 
activities  framed  within  the  guide  are  best  accomplished  by  more  senior,  experienced  training 
personnel. 


IV 


Utilization  and  Dissemination  of  Findings: 

The  guide  can  be  used  in  training  institutions  throughout  TRADOC  by  those  who  want  to 
develop  Prior  Knowledge  Assessments  to  predict  student  perfonnance,  which  in  turn  can  be  used 
to  identify  students  who  could  benefit  from  tailored  training.  It  is  designed  for  training 
developers,  course  managers,  experienced  instructors,  and  other  senior  level  training  personnel. 
In  addition,  it  is  most  applicable  for  use  in  technical  or  extended  training  courses,  portions  of 
courses,  or  specific  blocks  of  training  that  require  fonnal  standards  of  proficiency  as  students  in 
these  training  settings  generally  benefit  most  from  tailored  training.  Assessing  prior  knowledge 
in  these  training  situations  makes  it  easier  for  the  instructor  to  effectively  tailor  training  to  meet 
individual  student  needs.  Predicting  student  performance  in  other  Anny  training  situations 
where  the  intent  is  to  “familiarize”  students  with  information  might  be  less  beneficial.  Final 
copies  of  the  guide  and  this  report  will  be  provided  to  each  of  the  training  institutions  that 
provided  inputs  and  feedback  during  its  development.  Additionally,  copies  will  be  provided  to 
other  TRADOC  agencies  upon  request. 
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Prior  Knowledge  Assessment  Guide 


Background 

The  ultimate  goal  of  Army  instructors  is  to  deliver  the  best  possible  instructional 
experience  to  their  students  in  order  to  increase  learning  and  to  facilitate  the  maximum  transfer 
of  knowledge.  To  accomplish  that  goal,  an  instructor  needs  to  consider  individual  differences  in 
the  background  and  knowledge  of  students  in  a  class.  The  diversity  of  backgrounds  and 
knowledge  of  students  within  Anny  classrooms  can  be  substantial,  which  in  turn  warrants 
attention  to  individual  differences  that  impact  the  learning  process. 

Tutoring  is  often  cited  as  the  best  means  of  maximizing  learning  as  it  addresses 
individual  differences.  Bloom’s  (1984)  classic  article  on  tutoring  is  often  cited  as  support  for 
this.  However,  one-on-one  tutoring  is  not  logistically  feasible  in  Anny  classrooms.  Yet  some 
fonn  of  tailoring  instruction  to  students,  by  distinguishing,  at  a  minimum,  between  those  who 
will  struggle  with  the  material  and  those  who  will  easily  excel,  can  be  achieved.  In  FY09  the 
Army  Research  Institute  (ARI)  Research  Unit  at  Fort  Benning,  GA  started  a  multi-year  research 
program  on  tailored  training  with  the  ultimate  objective  of  developing  training  approaches  to 
address  critical  individual  differences  in  Soldiers’  backgrounds  and  skills  to  make  training  more 
effective,  meaningful  and  efficient. 

Initial  research  in  this  tailored  training  program  was  on  detennining  what  assessment 
instruments  or  measures  were  effective  in  predicting  differences  in  Soldier  performance  in  a 
variety  of  courses;  assessment  instruments  that  could  be  used  by  an  instructor  as  early  indicators 
of  Soldiers  who  would  need  assistance  and/or  Soldiers  who  would  excel  and  should  be 
challenged.  While  this  research  was  in  progress,  the  Anny’s  Training  and  Doctrine  Command 
(TRADOC)  published  the  Army  Learning  Concept  (TRADOC,  2011),  which  outlined  the  Army 
Learning  Model  (ALM).  One  of  the  principles  in  the  ALM  is  that  Anny  institutions  must 
account  for  students’  prior  knowledge  and  experiences  by  assessing  competencies  and 
appropriately  tailoring  learning  to  students.  No  further  information  was  cited  regarding  how  this 
objective  could  be  accomplished  or  how  to  assess  prior  knowledge  and  experiences. 

The  Prior  Knowledge  Assessment  Guide  presented  in  this  report  is  based  on  the  research 
efforts  and  lessons  learned  in  developing  predictors  of  Soldier  performance  emerging  from 
ARI’s  tailored  training  research  program.  This  research  has  consistently  shown  that  the  best 
predictor  is  an  assessment  of  relevant  prior  knowledge;  knowledge  which  is  specific  to  key 
concepts  and  practices  within  the  course  content.  The  guide  also  addresses  the  TRADOC ’s 
ALM  objectives  by  presenting  procedures  by  which  senior  instructors,  training  managers,  and 
course  developers  can  generate  measures  that  predict  or  estimate  Soldier  perfonnance  and  can 
assess  the  adequacy  of  the  measures  they  develop.  However,  the  guide  does  not  infonn  the  user 
how  to  tailor  training,  only  how  to  identify  students  who  may  benefit  from  some  form  of  tailored 
training.  It  should  also  be  noted  that  while  much  of  the  supporting  literature  uses  the  tenn  “test,” 
for  the  purposes  of  the  guide  and  this  report,  the  tenn  “assessment”  was  chosen  to  reduce 
potential  confusion  with  graded  events  and  other  training  metrics. 
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Predictors  of  Achievement  and  Performance 


The  body  of  research  on  identifying  predictive  measures  of  human  perfonnance  and/or 
achievement  is  large.  Often,  when  people  think  of  predictive  measures  they  first  think  of  using 
such  measures  for  personnel  selection  for  jobs  or  for  college  admission.  The  assumption  is  that 
these  measures  can  identify  individuals  who  will  perform  well  on  the  job  or  at  a  specific  college. 
The  military  itself  has  a  long  history  of  developing  measures  of  mental  ability  or  cognitive  skills 
for  personnel  selection;  beginning  with  World  War  I  (Zeidner  &  Drucker,  1988).  Efforts 
continue  to  the  present  to  develop  measures  that  will  provide  better  selection  tools  for  military 
branches  and  occupational  specialties  (e.g.,  Allen  &Young,  2012;  Heffner,  Campbell,  & 
Drasgow,  2011;  Howse  &  Damos  (2011);  Knapp  &  Tremble,  2007;  Zook,  1996).  A  variety  of 
variables  can  be  used  for  selection  purposes.  Common  variables  include  mental  or  cognitive 
ability,  prior  work  experience,  prior  knowledge,  past  performance,  acquired  skills,  physical 
ability  and  personality  traits  (Dakin  &  Annstrong,  1989,  U.S  Department  of  Labor,  2006). 

The  purpose  of  predictive  measures  for  tailoring  training  is  slightly  different,  in  that  you 
need  some  means  of  distinguishing  among  students  who  need  different  types  of  learning 
experiences  or  conditions  in  order  to  succeed  in  a  formal  training  setting.  One  of  the  issues 
associated  with  such  predictors  is  that  the  criterion,  achievement  in  a  classroom,  is  not  stable;  it 
is  dynamic,  and  what  an  individual  leams  changes  over  time  while  instruction  occurs.  Therefore, 
predicting  achievement  with  a  high  degree  of  accuracy  is  very  difficult.  Dyer  (2004)  in  a 
discussion  of  why  predictors  are  difficult  to  find  referred  to  Regain  and  Schneider’s  (1990) 
examination  of  Air  Traffic  Controller  selection.  Regain  and  Schneider  reported  that  an  extensive 
battery  of  tests  to  select  air  traffic  controllers  after  President  Reagan  decided  to  replace  striking 
controllers  did  not  work.  Only  2  percent  of  70,000  applicants  were  selected  and  about  half  of 
these  failed  the  training.  The  authors  attributed  this  to  two  factors.  One  was  poor  mapping 
between  the  predictor  and  criterion  measures  in  that  the  predictor  measures  pertained  to 
relatively  trivial  components  unrelated  to  the  criterion  test.  The  second  factor  was  ignoring 
“skill  plasticity”.  Research  on  skill  learning  clearly  shows  that  the  most  rapid  gains  in 
performance  occur  early  in  training,  and  then  only  gradually  increase  in  later  stages  of  training 
and  practice.  Thus  measures  that  predict  late  perfonnance  may  not  predict  early  performance. 
Regain  and  Schneider  stressed  the  importance  of  using  predictors  that  tap  the  cognitive  processes 
or  knowledge  required  of  the  criterion  task(s). 

Marksmanship  data  also  illustrate  the  concept  of  skill  plasticity.  While  Dyer  (1999) 
found  relatively  high  correlations  (0.50  to  0.67)  between  practice  record  fire  and  record  fire  for 
three  sighting  systems,  the  relationship  between  scores  on  one  sighting  system  and  another  were 
minimal.  Some  Soldiers  indicated  that  they  were  uncomfortable  in  using  some  of  the  sights, 
which  were  new  to  them,  at  this  particular  stage  in  training.  A  summary  of  correlations  among 
the  periods  in  basic  rifle  marksmanship  training  clearly  shows  that  marksmanship  skills  in  initial 
entry  training  are  not  stable  until  the  later  periods  of  instruction  which  involve  record  fire  (Dyer 
et  ah,  2012).  Rounds  to  group  and  zero,  measures  of  perfonnance  in  the  earliest  periods  of 
instruction,  did  not  conelate  with  each  other  or  with  later  measures  of  performance. 
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Using  Prior  Knowledge  as  a  Predictor  for  Tailoring  Training 

Clearly,  there  are  challenges  in  finding  measures  that  are  “good”  predictors  of  classroom 
perfonnance.  In  a  review  of  literature,  Schaefer  and  Dyer  (2012,  2013)  argued  that  the  best 
predictor  is  relevant  prior  knowledge.  This  conclusion  was  based  on  research  that  showed  prior 
knowledge  was  a  better  predictor  of  job  perfonnance  than  general  mental  ability  or  measures  of 
experience.  The  rationale  for  this  finding  was  that  “prior  knowledge  is  in  effect  the  combination 
of  capacity  to  learn  (general  mental  ability)  wed  to  the  opportunity  to  leam  (experience)” 
(Schaefer  &  Dyer,  2013,  p.  25).  Thus  experience  only  provides  an  opportunity  to  gain  skills  and 
knowledge,  but  does  not  guarantee  it  or  provide  that  the  appropriate  skills  and  knowledge  will  be 
obtained  or  retained.  Consistent  with  the  findings  by  Regain  and  Schneider  (1990),  Glaser 
(1984)  also  stressed  the  importance  of  domain-specific  prior  knowledge  and  skills  as  critical  to 
later  achievement  and  learning.  Relevance  of  prior  knowledge  is  cited  in  the  tutoring  literature 
as  well,  being  the  primary  means  by  which  tutors  could  individualize  their  instruction  and 
customize  and  elaborate  on  their  explanations  to  individual  students  (Wittmer,  Nuckles, 
Landmann,  &  Renkl,  2010). 

Most  of  the  research  on  using  measures  of  individual  differences  to  determine  the  best 
tailored  training  approaches  (e.g.,  aptitude-treatment  interactions  (Corno  &  Snow,  1986; 
Schaefer  &  Dyer,  2012,  2013)  has  been  conducted  in  public  school  or  experimental  settings. 
Aptitude-treatment  interaction  research  has  typically  not  been  conducted  extensively  with  adult 
students,  particularly  Soldiers.  Thus  it  was  important  to  determine  what  would  be  good 
predictors  in  Army  courses  to  detennine  if  the  findings  would  replicate  those  in  the  literature  and 
to  determine  lessons  learned  regarding  how  to  develop  and  apply  these  measures. 

Predictors  of  Performance  in  Army  Courses 

As  part  of  the  ARI  tailored  training  research  program,  predictors  of  course  performance 
were  developed  and  examined  for  six  efforts:  Warrant  Officer  Candidate  School,  Engineer 
Captains  Career  Course  (CCC),  Infantry  Advanced  Leader  Course,  Mechanical  Maintenance 
courses  for  the  Abrams  tank  and  Bradley  Fighting  Vehicle,  and  two  marksmanship  courses 
(Infantry  One  Station  Unit  Training,  and  Squad  Designated  Marksmanship  Course).  The  results 
and  lessons  learned  are  summarized  here  as  they  provide  the  foundation  for  the  Prior  Knowledge 
Assessment  Guide. 

The  first  effort  was  with  the  Warrant  Officer  Candidate  School  (Schaefer,  Bencaz,  Bush, 
&  Price,  2010).  Instructors  were  interviewed  to  help  identify  what  differentiated  high-  vs.  low- 
performing  students.  Three  characteristics  were  cited  by  instructors:  initiative,  attention  to 
detail,  and  metacognition.  Existing  measures  of  initiative  and  metacognition  were  used.  A 
special  measure  was  developed  to  assess  attention  to  detail.  In  addition,  the  demographic 
background  of  the  Soldiers  (e.g.,  education,  years  in  service)  was  obtained  as  well  as  details  on 
their  prior  life  experiences.  No  specific  prior  knowledge  instrument  was  used  in  this  research. 
The  criterion  measure  was  the  average  of  three  academic  tests  given  in  the  course. 

Some  of  the  experience  variables  related  to  the  criterion  measure,  but  not  strongly.  The 
individual  difference  variables  suggested  by  the  instructors  did  not  relate  to  the  criterion.  The 
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authors  felt  that  demographic  variables  still  held  promise  as  predictors,  but  that  future  efforts 
should  focus  on  direct  measures  of  prior  knowledge,  not  instructor’s  recommendations  of  student 
characteristics. 

With  the  Engineer  CCC  research  (Schaefer,  Blankenbeckler,  &  Lipinski,  2011),  the 
criterion  was  a  defensive  planning  test  given  as  an  integral  part  of  the  course.  Five  types  of 
predictors  were  used.  First,  the  small  group  instructors  projected  the  officers’  later  performance 
(top  25%,  middle  50%,  and  bottom  25%).  Second,  demographic  or  biographic  data  were 
obtained  (e.g.,  education  level,  prior  service  as  a  noncommissioned  officer,  deployment 
experience).  The  officers  also  provided  two  self-reports.  One  was  on  their  prior  experience 
relevant  to  defensive  planning  (training,  military  experience).  The  other  focused  on  their  self- 
confidence  in  executing  military  activities  related  to  defensive  planning. 

The  fifth  type  of  predictor  was  a  prior  knowledge  test  of  defensive  planning.  The  prior 
knowledge  test  was  based  on  several  factors  including  instructor  input  and  review.  It  was 
designed  to  test  more  than  facts.  It  also  focused  on  the  officers’  ability  to  use  information  and 
apply  principles  and  concepts,  placing  the  officer  in  the  role  of  a  task  force  Engineer.  The  test 
included  situational  descriptions,  and  samples  of  documents  which  an  officer  might  have  in  the 
field  such  as  tactical  diagrams,  photos  of  opposing  force  engineer  systems,  and  planning 
documents.  Incorrect  options  were  based  on  common  errors.  Perhaps  more  important,  was  that 
the  research  team  had  access  to  the  defensive  planning  criterion  measure.  Consequently,  they 
were  able  to  better  identify  relevant  prerequisite  knowledge  and  understandings  which  should 
have  been  gained  by  the  officers  in  prior  military  training,  specifically  the  required  Engineering 
Basic  Officer  Leader  Course. 

Findings  showed  that  for  officers  with  no  prior  enlisted  experience,  prior  knowledge 
alone  was  a  significant  predictor  (r  =  .45).  In  contrast,  for  officers  with  prior  enlisted  experience, 
there  were  no  significant  predictors  from  the  battery  of  predictors.  The  authors  speculated  on  the 
reasons  for  this  difference,  but  did  not  reach  a  definitive  explanation  of  the  findings. 

The  research  with  the  Infantry  Advanced  Leader  Course  (Schaefer,  Blankenbeckler,  & 
Brogden,  2011)  was  on  predicting  performance  in  two  different  blocks  of  instruction:  a  troop 
leading  procedures  (TLP)  test  and  a  land  navigation  field  exercise.  The  predictors  paralleled 
those  used  in  the  Schaefer,  Blankenbeckler,  &  Lipinski  (2011)  research  with  the  Engineer  CCC. 
Predictors  were  small  group  instructors’  prediction  of  perfonnance,  biographical  data,  personal 
prior  experience  in  the  two  domains  of  interest  (TLP  and  land  navigation),  and  prior  knowledge 
tests  relevant  to  each  criterion  of  interest. 

The  prior  knowledge  measure  for  TLP  included  items  on  TLP  necessary  for  an  offensive 
action,  information  on  how  a  student  would  prepare  to  brief  a  squad,  and  an  understanding  of  and 
ability  to  interpret  symbols  and  graphics.  The  land  navigation  prior  knowledge  test  was  written, 
whereas  the  criterion  was  field  performance.  The  written  test  included  maps,  photos,  etc.  Of 
particular  interest  was  that  the  land  navigation  test  included  items  on  advanced  orienteering 
skills,  knowledge  beyond  what  the  noncommissioned  officers  would  have  typically  learned 
during  their  previous  military  training  and  education.  Some  questions  were  included  which  had 
to  be  solved  without  a  map,  but  typically  are  solved  with  a  map.  The  intent  of  including  such 
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questions  was  an  effort  to  identify  individuals  who  excelled  on  the  land  navigation  performance 
criterion  test.  This  was  a  first  step  in  determining  how  to  develop  a  prior  knowledge  test  that 
could  be  used  to  identify  high  achievers. 

For  both  domains,  only  the  prior  knowledge  test  correlated  significantly  with  criterion 
perfonnance,  higher  for  TLP  (r  =  .40  for  TLP;  r  =  .28  for  land  navigation).  The  attempt  to 
identify  high  performers  on  the  land  navigation  field  test  was  only  moderately  successful  based 
on  an  item  analysis  of  the  most  difficult  items  on  the  prior  knowledge  test. 

Predictors  of  performance  in  two  vehicle  mechanical  maintenance  courses,  Abrams  tank 
and  Bradley  Fighting  Vehicle  were  examined  in  another  effort  (Cobb,  Schaefer,  Stallings, 
Blankenbeckler,  &  Wampler,  2014).  The  criterion  in  each  course  was  the  average  percentage  of 
“GOs”  received  on  a  series  of  tests  in  each  course.  Four  types  of  predictors  were  used.  One  was 
a  Soldier  questionnaire  tapping  relevant  prior  experience  with  electronics  and  vehicle 
maintenance.  The  General  Technical  (GT)  scores  from  the  Anned  Services  Vocational  Aptitude 
Battery  (ASVAB)  were  obtained  from  student  records.  Three  cognitive  measures  from  the 
Educational  Testing  Service  kit  (Ekstrom,  French,  Harman,  &  Derman,  1976)  were  selected: 
Following  Directions  (integrative  processes),  Building  Memory  (working,  visual  memory),  and 
Choosing  a  Path  (spatial  scanning).  These  had  some  face  validity  for  tapping  the  cognitive 
processes  involved  in  the  mechanics  training.  Prior  knowledge  of  basic  electrical  circuits  and 
symbols  was  also  assessed.  These  technical  prior  knowledge  items  were  developed  by  an 
experienced  instructor,  not  by  the  research  team  as  in  the  prior  ARI  research  efforts. 

For  each  course,  the  only  significant  correlate  with  the  criterion  measure  was  prior 
knowledge  (r  =  .22  for  Abrams  course;  r  =  .39  for  Bradley  course).  It  should  be  noted  that  prior 
knowledge  did  not  correlate  with  the  criterion  in  an  experimental  version  of  the  Bradley  course, 
and  the  correlations  among  the  predictor  measures  (experience,  GT  and  prior  knowledge)  were 
consistent  with  the  model  developed  by  Schmidt,  Hunter,  and  Outerbridge  (1986),  although  the 
magnitude  of  the  correlations  were  lower.  The  authors  noted  that  the  criterion,  based  on 
Go/NoGo  dichotomous  scores,  was  restricted  in  range,  which  typically  results  in  attenuated 
correlations. 

There  were  at  least  two  lessons  learned  from  the  vehicle  maintenance  effort.  The  first  is 
the  importance  of  having  a  criterion  measure  that  is  sensitive  to  individual  differences  in 
perfonnance  levels.  Go/NoGo  or  Pass/Fail  measures  are  not  sensitive  to  differences  in 
individual  performance.  The  second  lesson  learned  is  that  experienced  instructors  can  develop 
measures  of  relevant  prior  knowledge  based  on  their  experience  with  students  in  prior  courses. 
They  have  a  “good  understanding  of  the  basic  skills  that  are  most  relevant  to  critical  learning 
objectives  and  enable  students  to  excel  in  training”  (Cobb  et  ah,  2014,  p.  34).  They  also  know 
the  criteria  for  performance  in  the  course,  providing  them  with  insight  and  a  unique  perspective 
regarding  the  prerequisite  knowledge  requirements  for  their  course. 

The  last  effort  (Lipinski,  James,  &  Wampler,  2013)  focused  on  marksmanship  skills  with 
the  criterion  being  hands-on  perfonnance,  but  the  predictors  were  paper-pencil  tests.  With  this 
research  the  major  purpose  was  to  determine  whether  perfonnance  on  a  prior  knowledge  test  of 
marksmanship  added  any  predictive  power  beyond  that  from  simply  asking  if  Soldiers  had 
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shooting  experience  outside  of  the  military.  One  course  was  the  Squad  Designated 
Marksmanship  (SDM)  Course,  which  is  typically  attended  by  NCOs  who  have  several  years  of 
service  in  the  Anny.  In  this  case,  the  average  time  in  service  was  8  Vi  years.  The  other  course 
was  Basic  Rifle  Marksmanship  (BRM)  taken  by  Soldiers  in  Infantry  One-Station-Unit  Training 
(i.e.,  “basic”  training).  Thus  the  Soldier  samples  were  quite  different  in  terms  of  military 
marksmanship  experience.  However,  the  predictor  measures  were  the  same  for  each  sample: 
demographic  infonnation  on  marksmanship  experience  outside  the  military  and  a  prior 
knowledge  test  which  had  two  versions,  a  short  fonn  (nine  matching  questions)  and  a  long  fonn 
(the  nine  matching  questions  along  with  16  multiple-choice  questions).  The  test  required 
students  to  match  descriptive  definitions  with  doctrinal  tenns  and  to  indicate  their  understanding 
of  ballistics,  minutes  of  angle,  iron  and  optical  sight  use,  and  the  effects  of  wind  on  the  trajectory 
of  the  bullet.  Lastly,  because  of  differences  in  the  courses,  the  criterion  live-fire  measures  also 
differed.  The  authors  found  that  prior  knowledge  significantly  predicted  marksmanship 
performance  beyond  any  effects  of  self-reported  prior  shooting  experience.  However,  the 
relationship  was  much  stronger  for  Soldiers  in  the  SDM  course  (for  both  fonns  of  the  prior 
knowledge  test)  than  for  those  taking  BRM  (correlations  in  SDM  were  typically  greater  than  .30 
with  the  highest  being  .65  vs.  correlations  being  less  than  .25  in  BRM). 

There  are  three  lessons  learned  from  the  marksmanship  research  effort.  First,  a  prior 
knowledge  assessment  can  be  short  and  still  be  effective  in  predicting  later  performance. 

Second,  a  prior  knowledge  test  should  actually  assess  what  is  assumed  to  be  prior  knowledge.  A 
very  likely  reason  for  the  difference  in  the  two  samples  was  that  the  prior  knowledge  test  did  not, 
in  reality,  assess  prior  knowledge  for  the  Infantry  trainees,  as  it  covered  content  which  is  actually 
trained  in  basic  rifle  marksmanship,  advanced  rifle  marksmanship,  and  even  specialized  shooting 
courses.  A  comparison  of  the  means  on  the  test  for  the  two  groups  showed  that  the  Infantry 
trainees  scored  much  lower,  indicating  their  level  of  prior  knowledge  was  less  than  those 
enrolled  in  the  SDM  course.  Third,  the  results  support  prior  findings  regarding  prior  knowledge 
tests  being  better  predictors  than  questions  on  experience. 

Lessons  Learned  from  the  ARI  Research  on  Predicting  Performance  in  Army  Courses 

In  summary,  in  each  of  the  six  efforts  conducted  as  part  of  the  ARI’s  tailored  training 
research,  a  diverse  set  of  measures  was  used  as  potential  predictors  of  performance  in  a  variety 
of  Army  courses  (different  subject  matter)  and  with  Soldiers  of  differing  ranks  and  experience. 
Measures  of  relevant  prior  knowledge,  used  in  five  of  the  six  efforts,  were  the  only  consistent 
significant  correlate  of  course  perfonnance.  Paper-pencil  measures  of  prior  knowledge  also 
predicted  hands-on  performance,  although  one  would  expect  a  hands-on  prior  knowledge 
measure  to  be  a  higher  correlate.  Instructors’  estimates  of  student  perfonnance,  prior  military 
experience,  standardized  cognitive  measures,  and  non-cognitive  measures  did  not  predict.  It  is 
acknowledged  that  the  prior  knowledge  correlations  were  not  always  high,  but  they  were 
consistent.  We  knew  from  prior  research  (Dyer,  Wampler,  &  Blankenbeckler,  2011)  that 
instructors  did  not  frequently  use  formal  or  systematic  techniques  to  identify  students  with 
minimal  knowledge  or  those  with  much  prior  knowledge.  They  would  informally  query  some 
students  about  their  military  experience  in  order  to  identify  those  who  might  excel  in  the  course 
and  those  who  would  have  difficulty.  But  as  determined  in  the  prior  efforts,  systematic  measures 
of  military  experience  did  not  predict  well,  if  at  all. 
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In  all  but  one  of  the  ARI  efforts,  the  prior  knowledge  tests  were  developed  by  the 
research  team,  in  conjunction  with  or  reviewed  by  instructors  or  course  managers  since  the 
researchers  lacked  content  expertise.  On  the  other  hand,  while  experienced  instructors 
demonstrated  they  could  apply  their  expertise  to  identify  what  key  subject  areas  should  be  in  a 
prior  knowledge  test,  they  needed  significant  assistance  in  how  to  develop  “good”  tests,  as  most 
did  not  have  training  in  educational  measurement.  In  addition,  most  instructors  have  not  had  any 
statistical  training  on  how  to  relate  prior  knowledge  data  to  criterion  performance  data.  Both  test 
development  and  some  basic  statistical  skills  are  necessary  to  make  decisions  about  which 
students  might  benefit  most  from  tailored  training.  These  factors  necessitated  a  guide  be 
developed  providing  instructors  with  the  tools  needed  for  test  development  and  application. 

Enabling  instructors  to  develop  prior  knowledge  assessments  also  means  that  in-house 
resources  will  be  used.  A  research  team  does  not  need  to  be  “called  in”  nor  is  there  a  need  to 
search  the  literature  for  measures  or  possibly  pay  a  cost  for  a  generic  predictor  measure. 

Another  advantage  of  prior  knowledge  assessments  is  that  they  have  “face”  validity. 
Students  don’t  wonder  “Why  am  I  taking  this?”  and  an  instructor  can  easily  explain  the 
relationship  between  prior  knowledge  and  the  ease  of  learning  new  information  and  skills. 

The  research  also  showed  the  importance  of  using  criterion  measures  that  are  sensitive  to 
individual  differences.  Although  the  research  did  not  examine  the  extent  to  which  prior 
knowledge  assessments  predicted  course  performance  at  different  points  in  time,  the  plasticity  of 
skill  learning  was  acknowledged  in  the  guide.  One  effort  also  demonstrated  the  importance  of 
clearly  defining/conceptualizing  what  is  meant  by  “relevant  prior  knowledge”  in  a  course. 
Experienced  instructors  know  the  student  populations  and  their  prior  fonnal  military  training.  If 
a  prior  knowledge  assessment  is  based  on  knowledge  to  be  learned  in  the  course  or  learned  in 
future  courses,  then,  by  definition,  it  is  not  prior  knowledge  and  predictions  will  be  minimal  at 
best.  The  next  section  in  the  report  describes  the  concept  of  relevant  prior  knowledge  in  more 
detail. 

Assessing  Prior  Knowledge 

The  key  to  assessing  prior  knowledge  as  a  predictor  is  to  assess  relevant  prior  knowledge. 
In  general,  prior  knowledge,  as  it  relates  to  military  training,  is  any  knowledge  a  student  has 
prior  to  beginning  training.  Thus,  a  student’s  prior  knowledge  can  consist  of  knowledge  in 
many  different  knowledge  areas.  For  purposes  of  military  training,  a  knowledge  area  is  factual 
information  directly  related  to  some  particular  area  of  interest  or  requirement  in  a  training 
program  or  course,  as  well  as  knowledge  on  specific  equipment  and  skills  associated  with  that 
area.  However,  not  all  prior  knowledge  is  directly  relevant  to  the  specified  training  to  be 
conducted.  When  prior  knowledge  possessed  in  any  specific  knowledge  area  is  applicable  to 
enhancing  success  in  subsequent  training,  that  knowledge  is  considered  relevant.  An  instructor 
must  detennine  exactly  which  knowledge  areas  have  a  potential  influence  on  planned  training 
outcomes  and  if  prior  knowledge  within  those  areas  enhances  a  student’s  ability  to  perform  well 
during  training.  Clearly,  not  all  of  a  student’s  prior  knowledge  meets  these  criteria.  For 
example,  relevant  prior  knowledge  for  tracked  vehicle  mechanics  training  is  likely  different  from 
what  prior  knowledge  would  be  relevant  for  radar  systems  operator  training.  Also,  prior 
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knowledge  of  basic  mechanical  tools  and  how  they  are  used  is  important  for  training  in  a 
technical  mechanics  course,  while  knowledge  of  how  to  splint  a  broken  leg  or  interpret  compass 
readings  is  irrelevant  for  maintenance  tasks.  Instructors  must  understand  the  breadth  and  content 
of  different  knowledge  areas  as  they  relate  to  any  specific  training  course. 

In  the  guide  presented  in  this  report,  a  prior  knowledge  assessment  is  a  tool  that  measures 
students’  levels  of  relevant  knowledge,  rather  than  just  collecting  general  infonnation  on 
students’  backgrounds  and  experiences.  This  data  should  allow  users  to  forecast  students’ 
probable  performance  during  training. 

Prior  knowledge  assessments  differ  from  pretests.  Like  pretests,  prior  knowledge 
assessments  are  administered  prior  to,  or  at  the  start  of,  training  and  often  consist  of  paper-based 
or  hands-on  evaluations  of  student  knowledge  and/or  skills.  Pretests  however,  detennine  how 
much  knowledge  a  student  currently  possesses  of  the  course  content;  thus,  questions  on  pretests 
assess  knowledge  about  what  is  to  be  taught  in  the  course.  Also,  most  pretests  will  include  test 
items  identical  to  or  very  similar  to  those  on  a  “posttest”  or  “final  exam.”  On  the  other  hand, 
prior  knowledge  assessments  measure  associated  knowledge  areas  that  will  aid  students  in  their 
learning  experience,  but  do  not  directly  assess  knowledge  about  specific  material  or  content  to  be 
taught  in  the  course.  For  example,  prerequisite  knowledge  and  skills  are  considered  relevant 
prior  knowledge. 

Each  prior  knowledge  assessment  should  be  unique  to  the  course  or  block  of  instruction 
for  which  it  was  designed.  Most  often,  instructors  are  in  the  best  position  to  detennine  what 
specific  knowledge  areas  are  most  relevant  for  their  training  course  and  to  training  success. 

From  past  training,  they  are  aware  of  prior  knowledge  deficits  exhibited  by  students  which 
negatively  impact  learning  as  well  as  acquired  knowledge  areas  that  greatly  facilitate  learning. 
Once  a  detennination  of  what  knowledge  areas  are  relevant,  an  assessment  must  be  obtained  or 
developed  to  measure  the  extent  to  which  students  possess  that  knowledge.  Since  such 
assessments  most  often  do  not  exist,  they  must  be  developed.  In  the  case  of  vehicle  maintenance 
training,  development  of  an  assessment  would  likely  include  questions  in  related  knowledge 
areas  such  as  those  regarding  the  identification  and  use  of  common  vehicle  maintenance  tools.  It 
might  also  include  questions  addressing  applicable  content  in  other  relevant  knowledge  areas 
such  as  basic  knowledge  on  electricity  and  wiring. 

Problem  Definition 

The  overall  intent  of  this  effort  was  to  assist  Anny  instructors  in  identifying,  at  the  start 
of  training,  candidate  students  to  receive  some  type  of  planned  tailored  training.  The  problem 
facing  instructors  today  is  that  although  relevant  prior  knowledge  may  be  a  predictor  of  future 
classroom  perfonnance,  assessing  the  level  of  prior  knowledge  in  students  can  be  challenging. 
Consider  the  following: 

•  Prior  knowledge  can  help  predict  future  perfonnance  in  the  classroom; 

•  Prior  knowledge  must  be  relevant  to  the  training  conducted; 

•  Levels  of  relevant  prior  knowledge  in  individual  students  correlate  with  performance; 

•  An  assessment  is  required  to  identify  student  levels  of  relevant  prior  knowledge; 
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•  Assessments  must  be  unique  in  measuring  levels  of  prior  knowledge  relevant  to  the 
training  to  be  conducted; 

•  Unique  assessments  most  often  do  not  exist  and  must  be  developed; 

•  Currently  Army  course  managers,  training  developers,  and  other  academic  personnel  are 
largely  not  trained  on  how  to  develop  unique  prior  knowledge  assessments;  and 

•  Army  personnel  are  also  not  trained  on  how  to  apply  assessment  results  to  identify 
potentially  weak  and/or  strong  students. 

It  is  worth  noting  that  predicting  student  performance  in  some  Army  training  situations 
might  be  less  beneficial  than  in  others.  Specifically,  training  intended  primarily  to  “familiarize” 
students  with  information  and  overall  concepts  generally  does  not  rely  on  tailoring  to  achieve 
training  goals.  Therefore,  developing  prior  knowledge  assessments  for  general  familiarization 
training  would  not  be  worth  the  effort.  However,  students  in  more  technical  or  extended  training 
courses,  portions  of  courses,  or  specific  blocks  of  training  that  require  formal  standards  of 
proficiency  generally  benefit  most  from  tailored  training.  Assessing  prior  knowledge  in  these 
training  situations  would  make  it  much  easier  to  effectively  tailor  training  to  different  students. 


Development  of  the  Prior  Knowledge  Assessment  Guide 
Development  Method 

This  effort  was  conducted  in  order  to  produce  a  Prior  Knowledge  Assessment  Guide,  a 
copy  of  which  is  provided  in  the  Appendix.  Titled  “Guide  for  Developing  and  Using  Prior 
Knowledge  Assessments  to  Tailor  Training,”  the  guide  assists  academic  personnel  in  the  creation 
of  a  tool  to  help  identify,  at  the  start  of  training,  students  who  would  benefit  from  training  that  is 
tailored  to  their  needs.  The  specific  purpose  of  the  guide  is  to  assist  training  personnel  in 
creating  and  validating  prior  knowledge  assessments,  and  to  enable  them  to  use  the  results  to 
identify  students  who  would  benefit  from  tailored  training.  It  specifically  provides  guidance  on 
how  to  develop  assessments  in  order  to  identify: 

•  Students  who  may  need  additional  assistance  due  to  a  lack  of  or  minimal  amount  of 
relevant  prior  knowledge,  and/or 

•  Students  who  could  benefit  from  additional  challenges  due  to  their  familiarity  and 
understanding  of  requisite  knowledge. 

The  guide  needed  to  provide  sufficient  guidance  on  developing  prior  knowledge 
assessments,  determining  their  validity,  and  on  using  assessment  results  to  identify  students  who 
could  potentially  benefit  from  tailored  training.  It  is  acknowledged  that  the  “validation 
procedures”  cited  in  the  guide  are  not  equivalent  to  those  used  in  the  development  of  widely- 
used  tests.  From  the  viewpoint  of  a  psychometrician,  they  would  probably  be  considered 
necessary  but  not  sufficient. 

The  decision  was  made  to  develop  the  guide  with  the  idea  that  users  will  not  need  a 
detailed  background  on  why  prior  knowledge  assessments  should  be  used  for  predicting  student 
perfonnance.  Our  initial  assumption  was  that  users  would  already  have  made  the  decision  to 
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create  a  prior  knowledge  assessment  and  the  guide  would  simply  walk  them  through  how  to 
effectively  do  that.  Based  on  this,  it  was  detennined  that  only  limited  background  infonnation  in 
the  introductory  portion  of  the  guide  would  be  provided  in  order  to  set  the  conditions  for  the  rest 
of  the  guide. 

Development  of  the  guide  began  with  an  examination  of  available  research  efforts  on 
assessing  prior  knowledge.  The  literature  review  included  additional  information  available  from 
the  academic  community.  An  analysis  of  all  information  gathered  provided  a  basis  for  the 
upfront  portion  of  the  guide  which  identifies  how  prior  knowledge  assessments  can  be  used  and 
what  type  of  prior  knowledge  is  considered  relevant.  Additional  research  involved  identifying 
applicable  sources  to  be  used  as  guidelines  for  developing  assessment  questions.  Finally  an 
examination  of  potential  methods  for  use  in  validation  of  assessments  was  conducted  to 
detennine  the  most  practical  means  for  performing  correlation  coefficient  computations  for 
potential  developers. 

Guide  Content 

The  guide  was  developed  along  the  lines  of  what  users  would  need  to  know  to  develop 
and  validate  a  prior  knowledge  assessment.  Along  with  a  short  introduction  in  Chapter  1  that 
provides  some  basic  background  infonnation  and  sets  conditions  for  the  remainder  of  the  guide, 
it  was  determined  that  three  additional  chapters  would  be  required  to  explain  the  main  topics 
including  development,  validation,  and  use  of  prior  knowledge  assessments.  The  chapters  are 
sequenced  in  the  order  they  would  likely  be  used.  Supplemental  infonnation  and  developmental 
exercises  are  included  in  appendices. 

Chapter  1.  Chapter  1  provides  an  overview  of  prior  knowledge  assessments  and  lists 
key  terms  and  definitions  used  throughout  the  remainder  of  the  guide.  Key  terms  and  definitions 
are  presented  due  to  the  differences  in  tenn  definitions  found  throughout  literature  and  to  provide 
a  basis  for  users  unfamiliar  with  associated  tenns  and  concepts. 

Chapter  2.  Chapter  2  describes  how  to  develop  a  prior  knowledge  assessment  to  fit  the 
user’s  needs.  It  provides  a  detailed  step-by-step  set  of  instructions  for  developing  a  prior 
knowledge  assessment  tailored  to  a  specific  course  or  block  of  instruction.  The  five 
developmental  steps  are  shown  in  Table  1. 
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Table  1 

Process  for  Developing  a  Prior  Knowledge  Assessment 


Development  Steps 

Scope 

Step  1 

Determine  what 
instruction  will  include 
tailored  training 

•  What  instruction  will  be  tailored? 
o  Entire  training  course? 
o  Specific  blocks  of  instruction? 

Step  2 

Determine  which 
students  will  receive 
tailored  training 

•  What  type  of  tailored  training  is  envisioned? 

o  Additional  assistance  for  students  likely  to  struggle, 
o  Additional  challenges  for  students  likely  to  excel. 

•  How  will  the  results  of  the  assessment  be  used  to  tailor  training? 
o  Identify  students  with  little  or  no  relevant  prior  knowledge, 
o  Identify  students  with  much  relevant  prior  knowledge. 

•  What  detemiines  a  low/high  performer  during  training? 

Step  3 

Determine  what  prior 
knowledge  areas  to 
measure 

•  What  knowledge  areas  are  relevant  to  the  training? 

•  Will  these  knowledge  areas  affect  the  student’s  learning 
experience,  particularly  initial  learning? 

Step  4 

Identify  the  specific 
content  for  each  prior 
knowledge  area 

•  What  specific  content  will  be  assessed  in  each  knowledge  area? 
o  Identify  the  specific  content  to  assess  within  each  knowledge 
area;  and 

o  Identify  content  that  best  defines  desired  student  knowledge 
levels  and  understanding  of  selected  knowledge  areas. 

Step  5 

Develop  the  questions 
for  the  prior  knowledge 
assessment 

•  What  questions  should  be  asked? 

o  Use  infonnation  from  Step  4  for  content. 

•  What  type  of  assessment  will  be  administered? 
o  Hands-on,  paper-based. 

•  How  many  questions  for  each  knowledge  area? 

o  Emphasis/time  allotted  per  knowledge  area  during  training. 

•  What  question  formats  will  be  used? 

o  Multiple-choice,  True-False,  Matching. 

•  What  cognitive  level  of  understanding  will  be  sampled  by  the 
questions  in  each  knowledge  area? 

o  Knowledge,  comprehension,  application. 

•  How  difficult  should  be  the  questions  in  each  knowledge  area? 
o  Easy,  hard,  complex. 

•  How  will  the  assessment  be  scored? 

Each  of  the  developmental  steps  is  discussed  in  detail  with  examples  added  to  provide 
clarity  for  the  information  and  topics  addressed.  This  chapter  provides  the  infonnation  necessary 
for  users  to  understand  the  importance  of  designing  prior  knowledge  assessments  in  a  way  that 
helps  identify  the  students  targeted  and  around  knowledge  areas  that  are  relevant  to  the 
instructional  material.  It  further  provides  numerous  examples  of  questions  and  types  of 
questions  to  assist  users  in  fonnulating  questions  for  their  assessment.  The  five  steps  were 
constructed  in  a  logical,  sequential  manner  in  order  to  ensure  the  user  was  not  only  aware  of  the 
significance  of  the  actions  involved  in  each  step  but  that  each  action  was  accomplished  in  the 
proper  sequence.  The  following  list  identifies  the  reasoning  for  each  of  the  five  steps. 
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•  Step  1 .  Prior  knowledge  assessments  are  unique  to  the  instruction  to  be  delivered.  As 
such.  Step  1  has  users  identify  what  instruction  the  assessment  will  be  used  for  prior  to 
beginning  development. 

•  Step  2.  This  step  encourages  users  to  have  a  clear  understanding  of  what  they  intend  to 
accomplish  by  using  a  prior  knowledge  assessment  before  they  begin  development. 
Without  this  understanding  those  developing  the  assessment  will  find  it  hard  to  continue 
with  the  following  steps. 

•  Step  3.  The  selection  of  knowledge  areas  is  vitally  important  to  creating  an  applicable, 
valid  prior  knowledge  assessment.  Without  the  proper  identification  and  selection  of 
knowledge  areas,  the  assessment  will  not  be  valid  and  any  data  obtained  from  the 
assessment  will  not  identify  the  appropriate  students. 

•  Step  4.  Once  the  knowledge  areas  are  identified,  users  must  decide  what  infonnation 
from  each  area  would  best  represent  how  familiar  a  student  is  with  the  associated  facts, 
concepts,  and  skills  it  contains.  Since  knowledge  areas  can  sometimes  tend  to  be  broad 
in  nature,  developers  need  to  consider  what  information  within  each  area  is  potentially 
most  relevant  to  the  training  of  interest  before  beginning  to  formulate  assessment 
questions. 

•  Step  5.  The  final  step  involves  the  actual  development  of  questions  that  will  be  on  the 
assessment.  Personnel  developing  these  assessments  very  likely  have  never  had  any 
formal  training  on  test  development  or  formulation  of  test  questions.  In  that  light,  the 
discussion  for  this  step  provides  question  development  guidance  and  includes  examples 
of  poor  and  good  questions  while  highlighting  the  dos  and  don’ts  of  each  type  of  question 
and  the  associated  potential  development  issues. 

Chapter  3.  Chapter  3  describes  the  process  of  validating  a  prior  knowledge  assessment. 
The  validation  process  presented  involves  determining  the  degree  to  which  the  prior  knowledge 
assessment  predicts  (correlates  with)  the  course  criterion  measure.  It  includes  why  validation  is 
necessary,  what  information  is  needed  in  order  to  validate  an  assessment,  and  the  steps  required 
for  performing  validation.  The  chapter  begins  with  a  discussion  on  what  validation  means  and 
why  it  is  necessary.  This  chapter  also  includes  information  on  selection  of  the  criterion  measure 
and  the  issues  associated  with  that  choice  (e.g.,  when  it  is  administered,  the  sensitivity  of  the 
criterion).  This  discussion  was  included  to  reinforce  that  it  is  vitally  important  for  users  to 
understand  the  need  for  validating  an  assessment.  Without  that  understanding,  users  may  feel 
that  an  assessment  they  spent  hours  working  on  is  a  good  product  and  may  be  tempted  to  use  it 
without  validation.  Since  assessments  are  only  as  good  as  their  ability  to  predict  outcomes  to  an 
acceptable  degree,  using  assessments  without  validation  will  likely  not  highlight  the  students  it 
was  designed  to  identify. 

The  guide  also  emphasizes  that  predictions  involve  some  degree  of  error;  that  the 
relationship  between  prior  knowledge  assessment  scores  and  criterion  scores  is  not  perfect.  The 
degree  of  risk  in  predicting  who  will  do  well  in  a  course  and/or  who  will  have  difficulties  is 
reduced  as  the  strength  of  the  relationship  increases  between  the  prior  knowledge  assessment  and 
the  criterion  scores.  This  concept  was  integrated  in  the  guide  so  users  will  have  realistic 
expectations  about  their  predictions  for  individual  students. 
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It  is  likely  that  users  may  not  be  familiar  with  Excel  or  the  mathematics  that  underlie 
correlation  coefficients.  Since  validation  includes  both,  the  steps  for  validation  explain  in  detail 
the  process  of  using  these  tools  to  validate  an  assessment.  The  steps  provide  a  procedure  with 
sufficient  detail  to  allow  a  user  to  validate  an  assessment  even  without  a  complete  understanding 
of  the  theory  behind  the  mathematical  calculations.  Each  of  the  validation  steps,  shown  in  Table 
2,  is  explained  with  corresponding  screenshots  of  Excel  spreadsheet  procedures  for  data  input 
and  manipulation. 

Table  2 

Prior  Knowledge  Assessment  Validation  Steps 


Validation  Steps 

Step  1 

Administer  the  Prior  Knowledge  Assessment 

•  Administer  the  assessment; 

•  Instructor  does  not  see  the  results; 

•  Instructor  conducts  training  as  usual;  and 

•  Collect  criterion  scores  after  training. 

Step  2 

Obtain  and  Input  data  into  an  Excel  spreadsheet 

•  Obtain  student  names  and/or  roster  numbers,  prior  knowledge 
assessment  scores,  criterion  measurement  scores; 

•  Create  an  Excel  spreadsheet; 

•  Record  three  sets  of  data  in  the  spreadsheet  (name/roster 
number,  assessment  score,  criterion  score);  and 

•  Check  all  data  for  accuracy. 

Step  3 

Compute  a  correlation  coefficient 

•  Create  correlation  coefficient  formula  in  the  spreadsheet;  and 

•  Compute  the  correlation  coefficient. 

Step  4 

Evaluate  the  strength  of  the  relationship  between  the  two  scores 
•  Determine  if  the  relationship  is  strong  enough  to  support  using 
the  prior  knowledge  assessment  to  tailor  training. 

Step  5 

Plot  the  relationship  between  the  two  scores 

•  Create  scatter  plot  in  Excel  using  input  data;  and 

•  Examine  the  scatter  plot  for  outliers  and  strength  of  the 
relationship. 

The  Excel  screenshots,  such  as  the  one  in  Figure  1,  are  provided  for  clarity  in  the  guide 
with  the  assumption  that  all  users  may  not  be  equally  familiar  with  using  Excel  software  for  data 
management  and  analysis.  The  screenshots  and  accompanying  discussion  provide  easy  to  follow 
instructions  that  even  novice  users  can  understand.  Even  if  users  do  not  understand  the  concept 
behind  correlation  coefficients,  by  following  the  step-by-step  validation  process  provided  in  the 
guide,  they  can  still  properly  validate  an  assessment. 
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Figure  1.  Excel  spreadsheet  example. 


Chapter  4.  Chapter  4  focuses  on  using  and  revising  prior  knowledge  assessments.  Once 
validation  has  been  attempted  a  prior  knowledge  assessment  will  either  be  found  acceptable  for 
use  or  not.  In  either  case,  users  need  to  know  what  to  do  with  it  next.  This  chapter  explains 
both.  For  validated  assessments,  this  chapter  explains  how  to  establish  cut  points  for  identifying 
students  for  tailored  training  followed  by  administrative  procedures  for  using  the  assessment  in 
subsequent  training.  Procedures  for  identifying  students  who  may  need  help  as  well  as  those 
who  need  to  be  challenged  are  presented.  This  is  the  critical  application  of  the  prior  knowledge 
assessment  after  its  validation.  For  assessments  that  were  not  validated,  this  chapter  identifies 
possible  contributing  reasons  and  suggestions  for  revising  the  assessment  in  order  to  increase  the 
chance  of  it  being  validated. 
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Assessment  of  the  Guide 


Assessment  Method 

The  purpose  for  obtaining  feedback  on  the  Prior  Knowledge  Assessment  Guide  was  to 
gain,  from  a  potential  user  perspective,  comments  on  its  content,  organization,  clarity,  ease-of- 
use,  readability,  and  understandability.  Reviewers  were  asked  to  read  through  the  guide  from  the 
perspective  of  a  potential  user  while  focusing  on  whether  or  not  it  provides  clear  guidance  on 
developing,  administering,  and  validating  prior  knowledge  assessments. 

Pilot  Assessment 

An  initial  pilot  assessment  was  conducted  with  a  small  group  of  potential  users.  The 
purpose  of  the  pilot  assessment  was  to  obtain  feedback  on  the  guide  as  well  as  feedback  on  the 
assessment  materials  developed  to  record  participant  reactions  and  comments.  The  pilot 
assessment  was  conducted  with  four  instructors  from  the  Ordnance  Mechanical  Maintenance 
School,  Fort  Benning  GA,  which  provides  advanced  individual  training  for  Soldiers  in  Annor 
and  Bradley  maintenance  specialties.  Responses  were  overall  favorable  which  resulted  in  very 
minor  changes  to  the  guide  and  no  changes  to  the  assessment  materials. 

Revised  Guide  Review 

The  revised  guide  was  subsequently  provided  to  an  experienced  training  cadre  and 
personnel  assigned  to  the  Infantry  Mortar  Leader  Course,  Fort  Benning,  GA,  and  the  Warrant 
Officer  College,  Fort  Rucker,  AL.  All  personnel  (N  =  9)  who  provided  feedback  on  and 
recommendations  for  revising  the  guide  were  experienced  instructors,  training  developers,  course 
managers,  training  team  chiefs,  or  other  training  subject  matter  experts.  Thus,  each  reviewer 
represented  the  intended  population  of  potential  users  of  the  guide.  In  addition  to  obtaining 
written  feedback,  some  reviewers  (N=3)  were  available  for  an  infonnal  discussion  which 
provided  clarification  and  amplification  on  the  responses  of  all  reviewers  from  their  respective 
organizations. 

Assessment  Feedback  Form 

An  assessment  form  was  generated  to  solicit  feedback  from  supporting  reviewers.  The 
questions  were  designed  to  obtain  perspective  users’  impressions  of  the  guide  in  providing  the 
material  in  a  manner  that  allows  users  to  create  and  validate  a  prior  knowledge  assessment.  The 
request  for  feedback  did  not  require  or  ask  for  reviewers  to  create  a  prior  knowledge  assessment. 
The  intent  was  only  to  gain  potential  users’  impressions  of  whether  or  not  they  believed  they 
could  develop  and  validate  a  prior  knowledge  assessment  using  the  guide. 

Assessment  Feedback 

An  analysis  of  the  scores  for  each  of  the  feedback  questions  and  the  accompanying 
written  comments  indicated  that  feedback  could  generally  be  categorized  into  two  groups: 
foundational  or  core-level  instructors  and  senior  level  academic  personnel  including  senior 
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trainers,  training  managers,  and  other  senior  academic  personnel.  General  feedback  indicated 
that  while  the  guide  was  a  well  written,  informative  set  of  guidelines  for  creating  a  Prior 
Knowledge  Assessment,  it  included  some  concepts  that  were  challenging  to  understand  and 
implement  for  some  lesser  experienced  readers. 

Based  on  the  feedback  we  received,  junior  level  instructors  tended  to  be  overwhelmed 
with  the  amount  of  information  in  the  guide  and  with  some  of  the  unfamiliar  concepts,  such  as 
completing  an  assessment  validation.  Comments  stemming  from  follow-on  discussions  with 
these  indicated  they  generally  prefer  infonnation  in  the  fonn  of  short,  step-by-step  instructions 
for  completing  the  task  at  hand.  As  described  in  the  guide,  development  of  a  Prior  Knowledge 
Assessment  for  a  particular  course  requires  higher  level  reasoning  and  decision  making  on  the 
part  of  the  developer  in  order  to  create  an  assessment  for  validation.  Thus,  steps  within  the  guide 
include  the  rationale  for  key  decisions  with  an  explanation  of  what  must  be  accomplished. 
Decisions  regarding  assessment  content  and  structure  can  be  challenging  for  less  experienced 
personnel  but  ultimately  rest  in  the  training  developer’s  hands.  Many  junior  level  instructors 
appeared  reluctant  to  stray  from  the  familiarity  of  the  instructional  concepts  and  approaches  they 
have  been  taught.  In  addition,  verbal  feedback  from  follow-on  discussions  with  instructors  who 
participated  in  this  effort  also  indicated  that  junior  instructors  tended  to  base  their  perceptions  of 
likely  student  performance  on  personal  observations  alone. 

More  senior  personnel,  e.g.  course  managers  and  senior  instructors,  readily  saw  the  value 
in  the  guide  and  in  using  Prior  Knowledge  Assessments.  Their  response  to  the  guide  was  largely 
positive  with  virtually  no  major  issues  in  wording,  understanding,  organization,  and  content. 
Some  feedback  from  senior  personnel  acknowledged  that  they  believed  junior  instructors  may 
have  difficulty  in  using  the  guide  were  they  to  attempt  to  design  a  Prior  Knowledge  Assessment. 

Comments  suggested  two  specific  additions  to  the  guide.  First,  reviewers  recommended 
including  a  glossary  and  index  similar  to  what  would  be  found  in  a  text  book.  Second,  they 
recommended  adding  something  similar  to  a  quick  start  set  of  instructions  that  highlights  the 
main  steps  in  developing  a  prior  knowledge  assessment. 

Final  Revision 

Ultimately  the  feedback  we  gathered  resulted  in  two  changes  to  the  guide.  Initially  the 
guide  was  designed  for  use  by  instructors,  training  developers,  course  managers,  or  other  training 
personnel.  Correspondingly,  verbiage  in  the  guide  originally  used  the  term  “instructors”  to 
encompass  all  potential  users  including  instructors.  The  decision  was  made  to  narrow  the  focus 
of  potential  uses  to  senior  level  instructors  and  other  senior  training  personnel,  including  course 
managers  and  training  developers,  with  input  from  junior  level  instructors,  as  appropriate.  The 
tenn  “developer”  replaced  references  to  “instructor”  to  encompass  these  personnel  and  more 
closely  reflect  the  likely  potential  users  of  the  guide. 

Secondly  a  “Quick  Reference  Prior  Knowledge  Assessment  Development  and  Validation 
Flowchart”  was  developed  and  included  as  an  Appendix  in  the  guide.  The  updated  guide,  to 
include  flowchart,  is  attached  as  the  Appendix  to  this  report.  It  was  determined  that  the 
inclusion  of  the  flowchart  would  accomplish  the  main  intent  behind  the  suggestion  for  an  index 
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and  add  a  reference  for  users  to  assist  them  during  assessment  development.  While  an  index 
would  only  direct  a  user  to  locations  of  specific  words  within  the  guide,  the  flowchart  directs 
them  to  specific  concepts  and  steps  in  a  logical  sequence.  The  flowchart  begins  with  a 
developer’s  decision  to  create  a  Prior  Knowledge  Assessment  and  systematically  directs  the  user 
step-by-step  through  the  process.  Each  step  includes  the  page  and  paragraph  number  where 
information  regarding  that  step  can  be  found  in  the  guide.  A  separate  glossary  was  detennined 
not  to  be  necessary  as  a  “Key  Term  Definitions”  section  is  included  in  Chapter  1. 


Conclusions 

Any  typical  training  situation  will  likely  have  students  who  will  excel  in  training  as  well 
as  those  who  will  be  more  challenged  to  learn  the  content  in  the  time  available.  Either  of  these 
groups  could  benefit  from  some  type  of  tailored  training,  but  identifying  students  in  these  groups 
often  becomes  clear  only  at  some  point  during  or  at  the  completion  of  training.  In  order  to 
effectively  tailor  training  to  meet  students’  needs,  instructors  must  be  able  to  identify  their 
relevant  strengths  and  weaknesses  as  close  to  the  start  of  training  as  possible.  The  Prior 
Knowledge  Assessment  Guide  was  designed  to  help  identify  students  who  could  benefit  from 
tailored  training.  Instructions  in  the  guide  can  be  used  by  developers  to  assist  instructors  in 
establishing  a  basis  for  tailoring  training  in  Army  courses  through  the  use  of  prior  knowledge 
assessments. 

Collectively,  research  prescribes  that  prior  knowledge,  as  a  variable,  may  be  used  to 
predict  performance  as  it  has  in  its  roots  aspects  of  performance  related  skills,  mental  abilities, 
and  experiences.  In  contrast,  mental  ability  and  prior  experience,  either  independently  or 
combined,  have  been  shown  to  be  poor  predictors  of  later  perfonnance  (Dokko,  Wilk,  & 
Rothbard,  2009).  Recent  research  in  several  Army  courses  supports  this  finding  as  well. 
Although  prior  knowledge  has  been  shown  to  correlate  well  with  subsequent  student 
perfonnance,  using  prior  knowledge  as  a  predictive  measure  for  Army  training  applications 
requires  a  means  for  measuring  that  knowledge  in  individual  students.  Since  prior  knowledge  is 
often  unique  to  some  specific  course  content,  prior  knowledge  assessments  must  also  be  unique 
to  each  course.  Moreover,  since  these  assessments  do  not  often  exist,  they  must  be  developed  in 
order  to  measure  the  extent  to  which  students  possess  relevant  prior  knowledge.  Generally, 
Army  training  personnel  are  not  equipped  with  the  necessary  infonnation  and  skill  they  need  to 
create  and  validate  assessments  that  meet  their  needs.  The  intent  of  developing  this  guide  was  to 
provide  infonnation  that  would  allow  for  the  development  of  an  assessment  uniquely  tailored  to 
the  training  for  which  it  would  be  used. 

Feedback  on  the  ease  of  use,  clarity,  readability,  understandability  and  completeness  of 
the  guide  was  largely  positive,  while  also  indicating  that  the  personnel  most  likely  to  use  the 
guide  to  develop  assessments  would  be  senior  level  training  personnel.  While  junior  level 
instructors  can  be  a  great  resource  in  identifying  appropriate  knowledge  areas  and  content  for 
specific  assessment  questions,  they  are  not  likely  to  be  the  personnel  actually  developing  these 
assessments.  Senior  level  instructors,  course  managers,  training  developers,  or  other  senior  level 
training  personnel  will  generally  develop  the  assessments  to  assist  instructors. 
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The  guide  will  be  most  useful  in  training  situations  where  students  are  taught  technical 
procedures  that  require  fonnal  standards  of  proficiency  where  tailored  training  can  be  more 
appropriately  applied  to  meet  individual  student  needs.  Instructors  who  provide  training 
intended  primarily  to  “familiarize”  students  with  infonnation  will  find  the  guide  less  useful. 
Similarly,  higher  level  courses  that  do  not  teach  technical  skills  but  rather  seek  to  exercise 
students’  cognitive  skills,  such  as  assessing  situations  and  infonnation,  formulating  opinions,  and 
drawing  conclusions,  may  find  it  much  more  difficult  to  develop  an  effective  Prior  Knowledge 
Assessment.  However,  these  training  personnel  may  still  benefit  from  the  guide,  as  it  does 
describe  a  logical,  sequential  thought  process  for  evaluating  prior  knowledge  as  it  relates  to 
subsequent  student  perfonnance. 

This  guide  can  be  used  by  any  TRADOC  training  institution  wishing  either  to  develop 
Prior  Knowledge  Assessments  to  predict  student  perfonnance  or  to  gain  a  better  appreciation  of 
how  prior  knowledge  can  potentially  affect  training  in  their  classrooms.  This  guide  will  be  of 
most  benefit  to  training  developers,  course  managers,  and  other  senior  level  training  personnel 
who  are  seeking  a  step-by-step  set  of  instructions  on  how  to  develop,  validate,  and  apply  the 
results  from  a  Prior  Knowledge  Assessment  to  identify  students  who  would  benefit  from  planned 
tailored  training. 
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Preface 


This  guide  assists  training  personnel  in  identifying,  at  the  start  of  training,  students  who 
would  benefit  from  training  that  is  tailored  to  their  needs.  The  specific  purpose  of  this 
guide  is  to  assist  those  personnel  in  creating  prior  knowledge  assessments  by 
describing  how  to  develop  unique  questions  tailored  to  their  specific  courses'  content 
and  goals.  It  specifically  provides  guidance  on  how  to  develop  assessments  of  a 
student’s  relevant  prior  knowledge  in  order  to  identify: 

•  Students  who  may  need  additional  assistance  due  to  a  lack  of  or  minimal  amount 
of  relevant  prior  knowledge,  and/or 

•  Students  who  could  benefit  from  additional  challenges  due  to  their  familiarity  and 
understanding  of  requisite  knowledge. 

This  guide  focuses  on  the  assessment  of  relevant  prior  knowledge  because  such 
assessments  have  been  shown  to  predict  how  individual  students  will  perform  during 
training.  Assessing  prior  knowledge  is  more  reliable  than  other  techniques  that  might 
be  used  such  as  subjective  judgments,  personality  measurements,  or  demographic 
information.  Sufficient  guidance  is  provided  in  the  guide  to  enable  personnel  to  develop 
these  prior  knowledge  assessments,  to  determine  their  validity,  and  then  how  to  use  the 
assessment  results  to  identify  students  who  could  potentially  benefit  from  tailored 
training. 

This  guide  does  not  explain  how  to  tailor  training.  Instead  it  provides  a  starting  point  for 
those  who  want  to  tailor  their  training  by  providing  guidance  for  developing  assessments 
to  identify  which  students  would  most  likely  benefit  from  training  that  is  tailored  to  their 
level  of  prior  knowledge. 

Examples  used  within  this  guide  are  intended  as  illustrations  and  clarifications  of  the 
topics  discussed.  Rather  than  limiting  all  the  examples  to  a  single  content  area  or 
training  environment,  a  variety  of  examples  are  used  to  further  illustrate  specific  points 
and  reflect  different  training  contexts. 

The  intended  audience  for  this  guide  is  training  developers,  course  managers,  senior 
level  instructors  or  other  personnel  likely  to  develop  Prior  Knowledge  Assessments. 
Within  this  guide,  the  term  “developers”  is  used  to  encompass  all  potential  users. 
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Chapter  1.  Introduction 

This  guide  is  designed  to  help  you,  the  developer,  assist  instructors  in  establishing  a 
base  for  tailoring  training  in  Army  courses  through  the  use  of  prior  knowledge 
assessments.  Any  typical  training  situation  will  likely  have  students  who  will  excel  in 
training  as  well  as  those  who  will  be  more  challenged  to  learn  the  content  in  the  time 
available.  Either  of  these  groups  could  benefit  from  some  type  of  tailored  training,  but 
identifying  students  in  these  groups  often  becomes  clear  only  at  some  point  during  or  at 
the  completion  of  training.  In  order  to  effectively  tailor  training  to  meet  students’  needs, 
instructors  must  be  able  to  identify  their  relevant  strengths  and  weaknesses  as  close  to 
the  start  of  training  as  possible. 

Often,  experienced  instructors  begin  a  course  with  an  idea  of  how  they  plan  to  tailor 
training  but  are  challenged  to  determine,  with  any  significant  confidence,  which  students 
their  planned  tailoring  would  most  benefit.  For  example,  if  the  instructor  wants  to  tailor 
training  to  focus  on  the  needs  of  weaker  students,  tailoring  cannot  be  implemented 
without  knowing  which  students  should  receive  it.  To  help  make  that  determination,  an 
early  assessment  of  students’  prior  knowledge  can  give  the  instructor  some  idea  of 
those  most  likely  to  struggle  with  the  content.  Assessments  that  provide  insights  into 
students’  relevant  prior  knowledge  levels  can  help  predict  who  are  most  likely  to  perform 
either  above  or  below  average  during  the  course.  This  information  enhances  the 
instructor’s  ability  to  target  the  right  students  to  receive  the  tailored  training  envisioned. 

It  does  not  however,  provide  assistance  in  how  tailoring  should  be  accomplished. 
Instructors  must  decide  how  they  plan  to  implement  tailoring  to  the  identified  students 
as  a  variety  of  methods  may  be  used. 

It  is  worth  noting  at  this  point,  that  predicting  student  performance  in  some  Army  training 
situations  might  be  less  beneficial  than  in  others.  Specifically,  training  intended 
primarily  to  “familiarize”  students  with  information  and  overall  concepts  is  generally  not 
demanding  and  does  not  rely  on  tailoring  to  achieve  training  goals.  Therefore, 
developing  effective  prior  knowledge  assessments  for  general  familiarization  training 
would  not  be  worth  the  effort.  However,  students  in  more  technical  or  extended  training 
courses,  portions  of  courses,  or  specific  blocks  of  training  that  require  formal  standards 
of  proficiency  generally  benefit  most  from  tailored  training.  Assessing  prior  knowledge 
in  these  training  situations  would  make  it  much  easier  to  effectively  tailor  training  to 
different  students. 

1.1  Overview  of  Prior  Knowledge  Assessments 

In  this  guide,  a  knowledge  area  refers  to  factual  information  directly  related  to  some 
particular  area  of  interest  or  requirement  in  a  training  program  or  course,  as  well  as 
knowledge  on  specific  equipment  and  skills  associated  with  that  area.  As  an  example, 
knowledge  and  skills  relating  to  the  identification  and  use  of  a  set  of  basic  mechanical 
tools  could  be  described  as  one  knowledge  area,  while  knowledge  relating  to  maps  and 
map  reading  could  be  described  as  another  knowledge  area. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING 


1 


What  is  Relevant  Prior  Knowledge? 

Prior  knowledge,  as  it  relates  to  military  training,  is  any  knowledge  a  student  has  prior  to 
beginning  training.  When  the  prior  knowledge  possessed  in  any  specific  knowledge 
area  is  applicable  to  enhancing  success  in  subsequent  training,  that  knowledge  is 
considered  relevant.  At  some  point,  it  must  be  determined  exactly  what  knowledge 
areas  have  a  potential  influence  on  planned  training  outcomes  and  if  prior  knowledge 
within  those  areas  enhances  a  student’s  ability  to  perform  well  during  training.  Clearly, 
not  all  of  a  student’s  prior  knowledge  is  relevant  or  will  potentially  influence  later  training 
experiences  or  success.  More  than  one  knowledge  area  may  be  relevant  to  a  particular 
course  or  block  of  instruction.  For  example,  relevant  prior  knowledge  for  tracked  vehicle 
mechanics  training  is  likely  different  from  what  prior  knowledge  would  be  relevant  for 
radar  systems  operator  training.  As  the  developer  you,  along  with  the  instructor,  must 
understand  the  breadth  and  content  of  different  knowledge  areas  as  they  relate  to  any 
specific  course. 

In  order  for  prior  knowledge  to  predict  or  relate  to  later  performance,  there  must  be  a 
reasonable  conclusion  that  students  possessing  knowledge  in  those  areas  will  do  better 
in  training  as  a  result  of  having  that  knowledge.  As  an  example,  suppose  your  course  is 
preparing  to  teach  a  new  group  of  wheeled  vehicle  mechanics  how  to  perform 
maintenance  on  a  selected  vehicle.  Instructors  may  want  to  first  determine  which 
students  actually  know  and  understand  the  use  of  the  basic  mechanical  tools  required 
to  perform  those  operations.  Understanding  the  differences  between  and  the  use  of 
various  tools  is  relevant  prior  knowledge  and  necessary  in  order  to  perform  more 
complicated  maintenance  tasks.  Whether  or  not  students  can  properly  splint  a  broken 
leg  or  interpret  compass  readings  are  irrelevant  for  performing  core  maintenance  tasks. 
Without  a  basic  understanding  of  tools  and  their  use,  a  student  will  have  a  tough  time 
performing  even  basic  maintenance  procedures.  Therefore  an  assessment  would  likely 
include  questions  regarding  the  identification  and  use  of  common  vehicle  maintenance 
tools  for  that  specific  knowledge  area.  It  might  also  include  additional  questions 
addressing  applicable  content  in  other  relevant  knowledge  areas  such  as  basic 
knowledge  on  electricity  and  wiring.  Most  often,  training  developers,  course  managers, 
and  instructors  are  in  the  best  position  to  determine  what  specific  knowledge  areas  are 
most  relevant  to  training  success. 

What  is  a  Prior  Knowledge  Assessment? 

A  prior  knowledge  assessment  is  a  tool  that  measures  students’  levels  of  relevant 
knowledge,  rather  than  just  collecting  general  information  on  their  backgrounds  and 
experiences.  This  data  should  allow  you  to  forecast  students’  probable  performance 
during  training.  Prior  knowledge  assessments  differ  from  pretests.  Like  pretests,  prior 
knowledge  assessments  are  administered  prior  to,  or  at  the  start  of,  training  and  often 
consist  of  paper-based  or  hands-on  evaluations  of  student  knowledge  and/or  skills. 
Pretests  however,  determine  how  much  knowledge  a  student  currently  possesses  of 
course  content;  thus,  questions  on  pretests  assess  knowledge  about  what  is  to  be 
taught  in  the  course.  Also,  most  pretests  will  include  test  items  identical  to  or  very 
similar  to  those  on  a  “post  test”  or  “final  exam”.  On  the  other  hand,  prior  knowledge 
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assessments  measure  associated  knowledge  areas  that  will  aid  students  in  their 
learning  experience,  but  do  not  directly  assess  knowledge  about  specific  material  or 
content  to  be  taught  in  the  course.  For  example,  prerequisite  knowledge  and  skills  are 
considered  relevant  prior  knowledge. 

It  is  important  to  understand  that  using  an  assessment  to  measure  an  individual’s  prior 
knowledge  is  only  a  measurement  that  estimates  the  level  to  which  that  individual 
possesses  that  knowledge.  As  such,  assessments  are  subject  to  errors  both  in 
measuring  the  actual  level  of  knowledge  as  well  as  in  their  indications  of  future  student 
performance.  When  using  assessments  designed  for  predictive  purposes,  you  need  to 
be  aware  of  these  limitations  and  should  not  expect  a  measurement  or  prediction  to  be 
perfectly  accurate. 

Where  Do  I  Get  My  Prior  Knowledge  Assessment? 

A  prior  knowledge  assessment  is  specific  to  the  course  or  block  of  instruction  for  which 
it  was  designed.  It  is  not  a  general  test  of  knowledge.  Once  you  determine  what  prior 
knowledge  areas  are  relevant,  you  must  either  obtain  or  develop  the  appropriate 
assessment.  Since  such  assessments  typically  do  not  exist,  this  guide  helps  you 
develop  them.  The  instructions  in  Chapter  2  (Developing  Prior  Knowledge 
Assessments)  give  you  the  information  you  need  to  move  forward. 

What  Do  I  Do  With  My  Prior  Knowledge  Assessment? 

Once  a  prior  knowledge  assessment  is  developed,  the  next  step  is  to  determine  if  it 
accomplishes  the  purpose  for  which  it  was  designed.  To  accomplish  this,  the 
assessment  must  be  validated  to  determine  how  well  it  predicts  either  low  or  high 
performers  depending  on  its  intended  use.  Validation  occurs  through  an  analysis  of  the 
assessment  results  as  compared  to  a  criterion  indicating  actual  student  performance 
after  training.  This  analysis  indicates  how  well  the  results  on  the  prior  knowledge 
assessment  predicted  actual  course  performance.  Chapter  3,  Validating  Prior 
Knowledge  Assessments,  provides  details  on  how  to  accomplish  this  validation  along 
with  information  on  what  the  comparison  results  mean. 

Once  an  assessment  has  been  validated,  the  assessment  may  be  used  in  subsequent 
classes  to  assist  in  tailoring  training.  In  such  cases,  the  assessment  should  be 
administered  prior  to  or  at  the  start  of  training,  its  results  examined,  and  training  tailored 
to  the  identified  students  as  envisioned.  Chapter  4,  Using  and  Revising  Prior 
Knowledge  Assessments,  contains  information  on  how  to  administer  a  prior  knowledge 
assessment. 
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1.2  Key  Term  Definitions  For  This  Guide 

For  clarity  and  consistency,  the  following  terms  and  their  associated  definitions  are 
provided  as  they  apply  to  the  content  in  this  guide. 

•  Knowledge  Area  -  Knowledge  relating  to  some  particular  area  of  interest; 
contains  factual  information  as  well  as  knowledge  on  associated  equipment  and 
skills  in  that  area. 

•  Prior  Knowledge  -  Any  knowledge  a  student  has  prior  to  beginning  training. 

•  Relevant  Prior  Knowledge  -  Knowledge  a  student  has  prior  to  beginning 
training  in  areas  determined  to  have  a  positive  influence  on  associated  training 
outcomes;  knowledge  that  would  likely  enhance  a  student’s  ability  to  perform  well 
during  the  training. 

•  Prior  Knowledge  Assessment  -  A  tool  that  generates  data  quantifying  students’ 
levels  of  relevant  prior  knowledge;  administered  prior  to  or  at  the  start  of  training. 

•  Validation  -  A  process  for  determining  how  well  a  prior  knowledge  assessment 
predicts  later  performance  in  the  course. 

•  Criterion  -  A  measurement  of  actual  performance  after  training  as  related  to 
course  objectives. 

•  Performance  -  An  inclusive  term  encompassing  knowledge,  skill,  and  hands-on 
abilities  demonstrated  during  training. 

•  Correlation  -  A  statistical  analysis  determining  the  relationship  between  two 
variables,  which  in  this  case  are  scores  on  the  prior  knowledge  assessment  and 
actual  student  performance  as  indicated  by  a  criterion  measurement  after 
training. 


1.3  Information  Included  in  This  Guide 

The  remainder  of  the  information  included  in  this  guide  will  provide: 

•  Guidance  on  how  to  develop  a  prior  knowledge  assessment 

•  Guidance  on  how  to  confirm  if  a  prior  knowledge  assessment  meets  the  needs 
for  which  it  was  designed 

•  Additional  information  on  other  actions  pertaining  to  the  development  and 
administration  of  prior  knowledge  assessments 

•  Quick  Reference  Prior  Knowledge  Assessment  Development  and  Validation 
Flowchart  (Appendix  A) 

•  Sample  exercises  for  practice/assistance  in  associated  Excel  based  calculations 
(Appendix  C) 
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Chapter  2.  Developing  Prior 
Knowledge  Assessments 

Instructors  wishing  to  use  prior  knowledge  assessments  to  tailor  training  face  a  number 
of  challenges,  including  the  availability  of  the  assessments  themselves  and  the  time 
needed  to  conduct  an  assessment.  Prior  knowledge  assessments  are  generally  not 
available  and  must  be  developed  from  scratch.  Additionally,  the  time  needed  for 
administering  prior  knowledge  assessments  is  often  not  in  the  course  schedule. 
However,  properly  developed  assessments  can  be  administered  fairly  quickly  with 
minimal  disruption  of  training.  The  remainder  of  this  chapter  provides  you  as  the 
developer  assistance  in  creating  prior  knowledge  assessments  to  assist  instructors  in 
meeting  their  needs  of  student  identification  for  tailored  training. 

2.1  How  Do  I  Develop  a  Prior  Knowledge  Assessment? 

Using  a  structured  approach  will  help  streamline  the  process  and  ensure  the 
assessment  supports  the  instructor’s  needs.  The  steps  presented  in  Table  1  use  a 
backwards  planning  approach  for  development.  The  process  encourages  you  to  first 
understand  the  purpose  behind  the  use  of  the  assessment  results  in  order  to  help 
determine  what  to  assess  and  how  it  should  be  assessed. 

Table  1 

Process  for  Developing  a  Prior  Knowledge  Assessment 


Development  Steps 

Scope 

Step  1 

Determine  what 
instruction  will 
include  tailored 
training 

•  What  instruction  will  be  tailored? 
o  Entire  training  course? 
o  Specific  blocks  of  instruction? 

Step  2 

Determine  which 
students  will  receive 
tailored  training 

•  What  type  of  tailored  training  is  envisioned? 

o  Additional  assistance  for  students  likely  to  struggle; 
and 

o  Additional  challenges  for  students  likely  to  excel. 

•  How  will  the  results  of  the  assessment  be  used  to  tailor 
training? 

o  Identify  students  with  little  or  no  relevant  prior 
knowledge;  and 

o  Identify  students  with  much  relevant  prior  knowledge. 

•  What  determines  a  low/high  performer  during  training? 

Step  3 

Determine  what  prior 
knowledge  areas  to 
measure 

•  What  knowledge  areas  are  relevant  to  the  training? 

•  Will  these  knowledge  areas  affect  the  student’s  learning 
experience,  particularly  initial  learning? 
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Development  Steps 

Scope 

Step  4 

Identify  the  specific 
content  for  each  prior 
knowledge  area 

•  What  specific  content  will  be  assessed  in  each  knowledge 
area? 

o  Identify  the  specific  content  to  assess  within  each 
knowledge  area;  and 

o  Identify  content  that  best  defines  desired  student 
knowledge  levels  and  understanding  of  selected 
knowledge  areas. 

Step  5 

Develop  the 
questions  for  the 
prior  knowledge 
assessment 

•  What  questions  should  be  asked? 

o  Use  information  from  step  4  for  content. 

•  What  type  of  assessment  will  be  administered? 
o  Hands-on,  paper-based. 

•  How  many  questions  for  each  knowledge  area? 

o  Emphasis/time  allotted  per  knowledge  area  during 
training. 

•  What  question  formats  will  be  used? 

o  Multiple-choice,  true-false,  matching. 

•  What  cognitive  level  of  understanding  will  be  sampled  by 
the  questions  in  each  knowledge  area? 

o  Knowledge,  comprehension,  application. 

•  How  difficult  should  be  the  questions  in  each  knowledge 
area? 

o  Easy,  hard,  complex. 

•  How  will  the  assessment  be  scored? 

2.2  Step  1:  What  instruction  will  include  tailored  training? 

Tailoring  may  be  planned  for  an  entire  training  course  or  for  one  or  more  blocks  of 
instruction  within  a  course.  The  first  step  in  assessment  development  is  to  determine 
which  training  course  or  blocks  of  training  will  be  tailored.  Training  that  requires  higher 
levels  of  proficiency,  such  as  when  students  receive  a  certification,  when  their  skills 
impact  human  or  equipment  safety,  or  when  graduates  will  be  required  to  operate  high 
dollar  systems,  are  often  good  candidates  for  tailoring.  Less  demanding  training,  such 
as  when  the  intent  is  primarily  to  “familiarize”  students  with  information,  generally  is  not 
a  good  candidate  for  tailored  training.  Since  the  objective  in  these  situations  is  simply 
familiarization,  exact  knowledge  and  skills  are  less  important  than  in  training  requiring 
higher  levels  of  proficiency. 

With  the  assistance  of  the  instructors,  you  should  first  examine  your  training  to 
determine  which  courses  or  blocks  of  training  are  suited  for  tailored  training  and  if  the 
resources  are  available  to  implement  the  type  of  tailored  training  envisioned.  Since  the 
structure  of  training  courses  throughout  the  Army  is  diverse  in  nature,  you  will  have  to 
determine  not  only  in  which  course,  portion  of  a  course,  or  block  of  training  tailoring  will 
be  implemented,  but  also  when  the  tailoring  is  likely  to  take  place.  Often,  courses  are 
sequential  in  nature  and  early  learning  in  the  course  can  greatly  impact  later  learning. 

In  these  courses,  building  a  firm  foundation  of  core  concepts  early  in  the  course  is 
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essential  to  performing  well  in  the  remainder  of  the  course.  Tailoring  may  only  be 
desired  in  the  up-front  portion  of  the  course  to  build  a  sound  foundation  for  later 
learning.  Other  courses  are  structured  with  various  blocks  of  instruction  that  are  not 
dependent  on  each  other  to  achieve  desired  learning  outcomes.  In  these  courses, 
performance  in  one  block  may  have  little  impact  on  performance  in  subsequent  blocks. 
Tailoring  in  these  courses  may  only  be  desired  during  certain  blocks  of  instruction  due 
to  their  difficulty,  complexity,  or  resource  requirements. 


Advanced  Leader  Course  (ALC)  Training  Practical  Example 
Step  1  -  Determine  what  instruction  will  include  tailored  training 

Training  manager  SFC  Dan  Cooper  is  preparing  for  a  new  group  of  ALC 
students.  Instructors  are  considering  using  tailored  training  to  assist  with 
some  of  the  instruction.  Through  past  experience  he  knows  that  the  block  of 
instruction  on  Tactics  and  Operations  tends  to  show  a  wider  range  of 
performance  scores  among  students  when  compared  to  the  other  blocks.  In 
that  particular  block  of  training  he  has  noticed  that  some  students  tend  to 
perform  extremely  well,  while  others  have  a  much  harder  time.  His  past 
experience  tells  him  this  seems  to  be  due  to  some  students  having  certain 
prior  knowledge  that  aids  them  in  the  training,  while  others  do  not. 

SFC  Cooper  and  the  instructors  decide  to  tailor  training  only  for  the 
Tactics  and  Operations  block  of  instruction  based  on  the  assumption  that 
differences  in  prior  knowledge  are  consistently  affecting  student  performance 
in  this  block.  To  assist  in  determining  which  students  the  tailoring  should 
focus  on,  SFC  Cooper  decides  to  develop  a  prior  knowledge  assessment  to 
identify  student  prior  knowledge  of  material  the  instructors  believe  may  be 
impacting  student  performance. 


Note:  The  ALC  training  example  above  is  included  for  further  clarification  and 
practical  understanding.  This  practical  example  will  be  continued  after  the 
discussion  for  each  of  the  development  steps. 


2.3  Step  2:  Which  students  will  receive  tailored  training? 

The  key  to  producing  assessment  results  useful  for  predicting  individual  performance  is 
to  understand  how  the  results  will  be  used.  In  other  words,  you  must  first  determine 
which  students  will  be  the  focus  of  the  envisioned  tailored  training  in  order  to  make  sure 
the  assessment  identifies  those  individuals.  For  example,  is  the  intent  to  find  the  most 
probable  top  performers  in  order  to  provide  them  with  more  challenging  training,  is  it  to 
find  those  who  are  more  likely  to  struggle  with  the  training  in  order  to  provide  them 
additional  assistance,  or  both?  Regardless  of  how  the  predictions  may  be  used,  it  is 
important  to  make  that  determination  prior  to  developing  an  assessment. 
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For  example,  assume  there  are  two  instructors  teaching  the  same  block  of  training  to 
two  separate  groups  of  students.  You  are  planning  to  develop  a  prior  knowledge 
assessment  to  assist  the  instructors  in  tailoring  training.  For  each  instructor  you  begin 
with  Step  1  to  determine  which  type  of  students  the  instructors  want  to  focus  their  efforts 
on.  The  first  instructor  decides  to  provide  additional  challenges  for  the  high  achievers 
within  the  course  to  give  them  an  opportunity  to  increase  their  knowledge  and  skills. 

The  second  instructor  wants  to  know  which  students  are  most  likely  to  struggle  through 
the  course  without  extra  assistance.  Although  both  prior  knowledge  assessments  may 
contain  questions  within  the  same  knowledge  area,  the  focus  and  difficulty  of  the 
questions  will  likely  differ.  In  the  case  of  the  assessment  for  the  first  instructor,  a  series 
of  harder  or  complex  questions  may  help  identify  those  likely  to  excel.  Those  who  can 
correctly  answer  the  harder  questions  should  also  be  able  to  answer  lower  level 
questions,  therefore  making  those  questions  unnecessary.  In  the  case  of  the  second 
instructor,  the  desire  is  to  find  students  with  potential  weaknesses  on  basic  concepts  or 
knowledge  areas.  For  this  assessment  a  set  of  low-level,  basic  questions  may  be  all 
that  is  needed.  More  complex  questions  would  not  be  necessary  since  students  unable 
to  properly  answer  low-level  questions  would  most  likely  not  be  able  to  correctly  answer 
more  advanced  ones. 


ALC  Training  Practical  Example 
Step  2  -  Determine  which  students  will  receive  tailored  training 

Since  this  block  of  training  is  fundamental  for  leaders  as  they  continue 
to  advance  in  rank  and  position,  the  instructor  believes  that  it  is  important  for 
all  students  to  have  a  good  understanding  of  Tactics  and  Operations.  With 
this  in  mind,  he  decides  it  is  more  important  to  assist  those  students  who  tend 
to  have  difficulty  with  the  training  than  to  challenge  those  for  whom  the 
training  comes  easier.  Reflecting  on  his  previous  experiences  and  the 
performance  of  previous  classes,  he  decides  that  students  having  at  least  a 
basic  foundation  of  information  relevant  to  the  course  in  seem  to  have  fewer 
problems  during  training.  Therefore,  the  instructor  asks  SFC  Cooper  to 
develop  a  prior  knowledge  assessment  to  identify  those  students  who  are 
weaker  in  these  relevant  knowledge  areas.  His  intent  is  to  tailor  training  by 
providing  additional  assistance  early  in  the  course  and  outside  of  normal 
training  hours  to  these  particular  students  to  bring  them  up  to  a  level  that  will 
make  classroom  training  easier  and  more  meaningful. 


2.4  Step  3:  What  Prior  Knowledge  Areas  Should  I  Measure? 

As  can  be  seen  in  the  above  example,  a  critical  step  in  prior  knowledge  assessment 
development  is  determining  what  knowledge  should  be  measured.  This  guide  has 
already  stressed  that  relevant  prior  knowledge  must  be  identified.  Clearly,  not  all  prior 
knowledge  is  relevant  to  the  instruction.  But  how  do  you  identify  what  prior  knowledge 
is  relevant? 
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A  major  factor  to  discovering  what  knowledge  is  relevant  lies  in  a  thorough 
understanding  of  the  training  content.  You,  along  with  the  instructors,  should  examine 
the  course  content,  required  performance  outcomes,  and  past  students’  performance 
trends,  then  reflect  on  what  prior  knowledge  seems  to  best  assist  students  in  achieving 
the  learning  objectives  of  the  training.  Selection  should  be  based  on  whether  the  prior 
knowledge  allows  students  to  grasp  the  learning  content  quicker  and  at  a  higher  level 
than  those  without  that  knowledge.  As  discussed  in  Chapter  1 ,  an  assessment  of  prior 
knowledge  is  distinctly  different  from  a  pretest  in  that  a  pretest  focuses  on  content 
taught  during  the  course,  and  most  likely  tested  in  the  course.  The  focus  of  a  prior 
knowledge  assessment  is  on  core  knowledge  and  skills  not  explicitly  taught  in  the 
course,  but  still  critical  in  that  they  enable  quicker  and/or  higher  learning  in  the  course. 

A  second  factor  is  to  leverage  the  instructor’s  experience  in  teaching  students  enrolled 
in  the  course.  Typically,  experienced  training  personnel,  over  time,  discover  what  prior 
knowledge  students  need  to  progress  at  or  above  the  desired  rate  within  the  course  and 
the  consequences  when  this  knowledge  base  is  insufficient.  Similarly,  instructors  are 
often  aware  of  what  additional  prior  knowledge  gives  students  an  advantage  in  the 
course. 

Some  training  courses  have  defined  prerequisites  and  their  instructors  expect  a  certain 
level  of  competency  in  those  areas  from  students  prior  to  attending  the  course.  When 
knowledge-related  prerequisites  are  specified,  it  is  because  the  knowledge  previously 
gained  serves  as  a  necessary  basis  for  obtaining  additional  related  knowledge  (e.g., 
completion  of  a  lower-level  course).  The  same  concepts  apply  in  the  vast  number  of 
courses  that  do  not  have  readily  defined  knowledge  related  prerequisites.  Students 
who  possess  some  level  of  knowledge  in  relevant  knowledge  areas  tend  to  excel  in  their 
learning  and,  alternatively,  those  unfamiliar  with  those  knowledge  areas  often  have 
more  trouble  learning  the  content  being  taught. 

For  example,  consider  new  Soldiers  attending  a  tracked  vehicle  mechanics  course. 

They  will  be  learning  how  to  change  broken  components,  service  vehicles,  and 
troubleshoot  a  variety  of  problems.  Assume  there  are  no  prerequisites  for  this  course. 
Students  coming  from  a  variety  of  backgrounds  and  experiences  will  soon  arrive  ready 
to  be  taught.  From  prior  experience,  the  instructor  knows  that  within  this  diverse  group 
of  students,  some  will  possess  knowledge  that  will  assist  them  in  the  course  and  some 
will  not.  This  instructor  decides  he  wants  to  use  a  prior  knowledge  assessment  to  help 
identify  those  students  that  may  have  difficulty  in  the  course  in  order  to  provide  them 
with  some  tailored  training  to  assist  them.  The  instructor  is  in  the  best  position  to  assist 
you  in  determining  what  knowledge  areas  are  relevant  to  the  training  content  and  if  prior 
knowledge  in  those  areas  will  likely  impact  learning. 

This  instructor  identifies  two  areas  that  should  be  examined  in  the  prior  knowledge 
assessment:  common  mechanic’s  tools  used  in  general  maintenance  of  vehicles  and 
basic  knowledge  of  electricity  and  wiring.  Since  a  large  portion  of  training  builds  on 
these  two  knowledge  areas  the  instructor  deems  them  as  relevant  to  the  training  and 
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decides  to  have  you  design  an  assessment  that  includes  questions  in  these  areas. 
Since  the  intent  is  to  tailor  training  to  students  who  may  require  assistance  during 
training,  the  questions  are  designed  to  discover  which  students  do  not  have  at  least  a 
low  level,  basic  knowledge  of  each  of  these  areas. 


ALC  Training  Practical  Example 
Step  3  -  Determine  what  prior  knowledge  areas  to  measure 

As  SFC  Cooper  prepares  to  develop  the  prior  knowledge  assessment, 
his  instructor  considers  what  specific  prior  knowledge  he  believes  aids 
students  in  the  Tactics  and  Operations  block  of  instruction.  The  instructor 
knows  that  much  of  the  training  centers  on  planning  tactical  operations  and 
subsequently  producing  operations  orders  to  support  them.  As  a  former 
Operations  NCO  and  instructor  for  the  past  two  years,  the  instructor  is  well 
versed  in  all  aspects  of  tactical  operations  and  knows  what  knowledge  is 
required  to  plan  them  along  with  their  supporting  documents.  His  experience 
both  outside  and  inside  the  classroom  leads  him  to  conclude  that  there  are 
three  knowledge  areas  associated  with  the  Tactics  and  Operations  block  of 
instruction  in  which  he  considers  essential  to  success  in  training.  These 
areas  include  basic  knowledge  associated  with  map  reading  (e.g.,  plotting  grid 
coordinates,  recognizing  terrain  features  on  the  map,  interpreting  imagery  and 
relating  it  to  the  map),  graphic  symbols  (e.g.,  overlays,  control  measures),  and 
creating  operations  orders  (e.g.,  organization  of  information,  content  of  each 
paragraph).  His  previous  teaching  experience  has  shown  him  that  without  a 
good  basis  of  knowledge  in  these  areas  students  tend  to  struggle  and  quickly 
fall  behind  other  students  in  training. 


2.5  Step  4:  How  Do  I  Identify  the  Specific  Content  for  Each 
Knowledge  Area? 

Using  the  prior  knowledge  areas  identified  in  step  3,  you  and  the  instructor  must  now 
determine  what  specific  content  within  those  areas  will  be  used  to  generate  questions  in 
the  assessment.  Since  knowledge  areas  can  sometimes  tend  to  be  broad  in  nature, 
consideration  should  be  given  to  determining  what  information  within  each  knowledge 
area  is  potentially  most  relevant  to  the  training  of  interest.  Usually  large  knowledge 
areas  consist  of  several  smaller,  more  generalized  knowledge  areas  in  which  some 
areas  are  potentially  more  relevant  than  others.  In  most  cases  it  would  be  impractical  to 
develop  a  comprehensive  assessment  covering  all  aspects  of  each  knowledge  area. 

The  objective  should  be  to  select  portions  of  the  main  knowledge  area  that  are  most 
relevant  to  the  training  and  that  would  provide  the  best  value  for  determining  whether  or 
not  a  student  has  prior  knowledge  within  that  area.  You  should  consider  asking  yourself 
questions  such  as: 
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•  “What  is  important  within  this  knowledge  area  that  makes  it  relevant  to  the 
training  of  interest?” 

•  “What  is  hard  about  the  training  as  it  relates  to  this  knowledge  area  that  makes  it 
difficult  for  some  students?” 

•  “What  prior  knowledge  do  higher  performing  students  consistently  have  that  help 
them  perform  well?” 

In  the  maintenance  example  discussed  in  step  3,  an  instructor  identified  two  knowledge 
areas  including  basic  tools  used  in  repairing  vehicles  and  basic  knowledge  of  electricity 
and  wiring.  Specific  content  within  those  two  areas  now  needs  to  be  determined. 
Identification  and  use  of  basic  tools  can  consist  of  a  vast  amount  of  knowledge. 
However,  it  is  not  reasonable  to  expect  students  to  arrive  for  training  already  completely 
familiar  with  all  the  tools  they  will  use  during  training.  Thus,  the  instructor  focuses  on 
basic  tools  commonly  used  in  a  number  of  repair  and  maintenance  settings  that  higher 
performing  students  seem  to  be  familiar  with  at  the  start  of  training.  For  example,  the 
instructor  determines  that  these  students  seem  well  versed  in  using  different  types  of 
hammers,  screwdrivers,  wrenches  (both  standard  and  metric  sizes),  different  types  of 
calibration  tools,  and  some  types  of  basic  power  tools.  In  order  to  determine  how 
familiar  a  student  is  with  tools,  either  a  sampling  from  each  of  these  categories  or  from 
the  most  common  or  most  used  categories  may  be  adequate.  Since  a  majority  of 
vehicle  mechanical  work  is  accomplished  using  fairly  common  hand  tools,  questions 
involving  those  items  should  indicate  whether  or  not  a  student  knows  what  they  are  and 
how  to  use  them.  Simple  questions  that  determine  if  students  know  the  difference 
between  a  flat  tip  and  a  Phillips  (or  cross  tip)  screwdriver,  between  standard  and  metric 
wrenches,  or  between  a  1/4”  and  a  1/2”  drive  socket  set  may  provide  the  instructor  the 
desired  information.  Similarly,  questions  such  as,  “Which  way  would  you  turn  a 
standard  bolt  to  tighten  it?”,  or  “From  a  group  of  tools,  select  the  appropriate  tool  to 
remove  the  plate  in  the  diagram  below”,  would  provide  additional  information  on  tool 
use.  These  types  of  basic  questions  regarding  tools  would  provide  the  instructor  with 
an  idea  of  students’  prior  knowledge  of  tools.  More  difficult  questions,  such  as  on  the 
use  of  a  calibration  wrench,  may  not  be  as  useful  if  the  intent  is  only  to  determine 
students’  basic  familiarity  with  tools.  A  similar  process  could  be  used  to  determine  the 
most  relevant  areas  and  content  within  the  knowledge  area  of  electricity  and  wiring  to 
the  training  and  develop  questions  to  sample  those  areas. 
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ALC  Training  Practical  Example 
Step  4  -  Identify  the  specific  content  for  each  knowledge  area 

SFC  Cooper  now  turns  his  attention  to  the  specific  content  to  be 
included  in  the  assessment.  He  considers  what  material  within  each  of  the 
knowledge  areas  of  map  reading,  graphic  symbols,  and  operations  orders 
should  be  used  to  develop  the  questions  for  the  assessment.  In  the 
knowledge  area  of  map  reading  SFC  Cooper  and  the  instructor  decide  there 
are  four  basic  areas  of  knowledge.  Those  include  (1)  map  legend,  (2) 
directions  and  azimuths,  (3)  scales,  grids,  and  distances,  and  (4)  terrain 
features. 

Through  experience  the  instructor  has  found  that  virtually  all  students 
are  generally  familiar  with  a  legend  and  terrain  features  on  maps  or  can  learn 
that  information  very  rapidly.  Those  less  familiar  with  working  with  maps 
often  have  the  most  difficulty  with  directions  and  azimuths  and  with  scales, 
grids,  and  distances.  The  instructor  decides  he  only  wants  to  include 
questions  in  those  two  areas  on  the  assessment.  In  the  knowledge  area  of 
graphic  symbols,  he  decides  he  wants  to  include  questions  regarding 
commonly  used  unit  symbols  and  graphic  control  measures.  In  the  last 
knowledge  area  of  operations  orders,  he  knows  that  although  his  students  will 
receive  training  on  developing  portions  of  an  operations  order,  he  has  found 
that  those  familiar  with  basic  aspects  of  operations  orders  tend  to  struggle 
less  during  the  instruction.  He  has  determined  that  in-depth  knowledge  of  all 
sections  of  an  operations  order  is  not  required  prior  to  training.  Therefore,  he 
decides  to  include  basic  questions  that  focus  on  identifying  the  components 
of  operations  orders  and  describing  what  they  are  used  for  to  help  determine 
which  students  lack  this  basic  understanding. 


2.6  Step  5:  How  Do  I  Develop  the  Questions? 

Typically  the  most  time  consuming  portion  of  assessment  development  is  the 
construction  of  the  assessment  questions.  Developing  good  objective  questions  that 
properly  assess  the  appropriate  content  is  a  challenging  process  that  can  take 
considerable  time  to  complete.  While  true-false  and  matching  questions  can  be 
relatively  simple  to  construct,  well  thought  out  multiple-choice  questions  can  be  more 
difficult  and  time  consuming.  While  you  may  choose  to  use  any  question  format  that 
meets  your  needs,  the  following  paragraphs  include  several  things  you  will  want  to  keep 
in  mind  during  question  development. 

Assessment  Type 

The  most  common  types  of  assessments  you  are  likely  to  use  will  be  hands-on,  paper- 
based,  or  a  combination  of  the  two.  Hands-on  assessments  are  administered  to 
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students  in  a  way  that  provides  them  the  opportunity  to  perform  a  task  or  procedure, 
often  with  some  associated  equipment  or  material  while  an  observer  scores 
performance.  Paper-based  assessments  consist  of  written  questions  students  are 
expected  to  answer. 

Hands-On  Assessments 

While  hands-on  assessments  can  be  worthwhile,  they  also  tend  to  be  time  consuming 
both  in  preparation  and  execution.  You  should  take  into  account  the  time  and  resources 
available  for  set-up  and  administration  of  hands-on  assessments  to  determine  if  they 
are  feasible  for  your  situation.  Generally,  each  student  would  be  observed  individually 
which  could  require  many  observers  and  lots  of  equipment  in  order  to  complete  the 
assessment  in  a  reasonable  amount  of  time.  Hands-on  assessments  can  be  extremely 
diverse  in  makeup  and  administration  depending  on  the  subject  matter  covered  and  the 
methods  used  to  determine  the  levels  of  prior  knowledge  being  assessed.  The 
uniqueness  of  hands-on  assessment  methods  will  require  you  to  determine  how  levels 
of  prior  knowledge  will  be  measured  objectively.  It  may  be  worthwhile  for  you,  with  the 
assistance  of  the  instructor,  to  first  determine  the  questions  you  wish  to  be  answered, 
such  as  those  asked  in  paper-based  assessments,  and  then  determine  how  those 
questions  can  be  transformed  into  a  hands-on  assessment.  The  assessment  must  be 
structured  to  yield  an  “objective”,  verifiable  score,  not  a  “subjective”  opinion  regarding 
the  quality  of  a  student’s  hands-on  performance.  Other  considerations  such  as  the 
number  of  knowledge  areas  to  include  and  the  length  of  the  assessment  will  also  need 
to  be  addressed. 

You  should  keep  in  mind  the  following  key  points  if  considering  using  a  hands-on  prior 
knowledge  assessment. 

•  Hands-on  assessments  are  most  often  designed  to  be  performance  oriented 
requiring  application  of  knowledge  as  well  as  factual  knowledge.  Application 
of  knowledge  concerns  a  higher  level  of  cognitive  understanding  rather  than 
simply  knowing  facts.  Although  hands-on  assessments  can  be  carefully 
designed  to  assess  lower  cognitive  levels,  assessing  a  student’s  knowledge 
of  facts  is  often  easiest  using  paper-based  assessments.  (Assessing  student 
understanding  using  different  cognitive  levels  will  be  discussed  later  in  this 
chapter.) 

•  Hands-on  assessments  should  be  designed  in  a  way  that  allows  for  an 
objective  assessment  using  a  continuous  numerical  measurement,  such  as  a 
percentage  of  correct  answers,  for  scoring.  A  single  dichotomous  score, 
such  a  “Go/No-Go”  or  “Pass/Fail”,  will  not  give  the  level  of  detail  needed  for  a 
prior  knowledge  assessment.  While  dichotomous  scores  are  applicable  for 
subsequent  measurements  of  performance,  which  will  be  discussed  later, 
they  should  be  avoided  if  possible  when  designing  prior  knowledge 
assessments.  In  order  to  objectively  score  a  hands-on  assessment  using 
continuous  numerical  measurements,  some  type  of  detailed  checklist  would 
have  to  be  developed.  The  checklist  would  be  used  to  score  individual 
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student  performance  which  in  turn  could  be  converted  to  an  assessment 
score. 

•  The  length  of  a  hands-on  assessment  is  directly  related  to  how  long  it  takes 
for  a  student  to  perform  the  tasks  as  well  as  how  many  students  can  be 
assessed  simultaneously.  When  using  hands-on  assessments,  you  may  find 
that  a  limitation  in  time  available  for  administration  results  in  a  limitation  of  the 
amount  of  knowledge  that  can  be  assessed. 

•  Hands-on  assessments  most  often  require  additional  resources  beyond  those 
needed  for  paper-based  assessments.  These  additional  resources  may 
include: 

o  Equipment  -  Hands-on  assessments  will  require  some  type  of 
equipment  and/or  other  materials  which  must  be  collected  for  use. 
o  Set  up  time  -  A  hands-on  assessment  requires  additional  time  to  set 
up  equipment. 

o  Administration  time  -  Additional  administration  time  is  often  required 
as  most  often  all  students  cannot  be  assessed  at  the  same  time.  Most 
often  one  observer  per  student  is  required  during  assessment, 
o  Observers  -  To  reduce  administration  time  multiple  observers  are 
often  required  in  order  for  multiple  students  to  be  observed  at  the 
same  time. 

Although  hands-on  assessments  can  be  applicable  for  determining  a  student’s  level  of 
prior  knowledge,  paper-based  assessments  tend  to  be  an  easier  and  more  efficient 
solution.  While  the  remainder  of  this  guide  discusses  paper-based  assessments,  much 
of  the  information  and  concepts  may  also  be  applied  to  hands-on  assessments. 

Paper-based  Assessments 

Paper-based  assessments  tend  to  require  fewer  resources,  be  less  time  consuming  to 
prepare,  and  are  easier  to  administer  than  hands-on  assessments.  Much  like  written 
tests  and  knowledge  checks  during  training,  written  assessments  usually  consist  of 
questions  developed  in  different  formats  for  students  to  answer.  They  also  tend  to  be 
simple  to  score  and  contain  clearly  correct  responses,  rather  than  depending  on  more 
subjective  assessments  of  hands-on  task  performance.  These  factors  make 
comparison  for  validation  and  predictions  easier.  The  challenge  with  paper-based 
assessments  is  in  the  development  of  questions  that  provide  a  good  sample  of  a 
student’s  level  of  prior  knowledge  in  the  areas  being  assessed. 

Question  Format  -  Paper-based 

The  time  needed  to  complete  a  paper-based  assessment  depends  largely  on  the 
number  of  questions,  the  question  formats,  and  the  complexity  of  the  questions. 
Logically,  more  questions  and  more  complex  questions  take  longer  for  a  student  to 
answer  than  fewer  and  simpler  questions.  In  determining  how  many  questions  to  ask 
for  any  particular  knowledge  area,  you  should  keep  in  mind  the  total  administration  time 
required  to  allow  students  a  reasonable  opportunity  to  answer  all  of  the  questions. 
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Table  2  provides  some  general  guidelines  to  assist  you  in  determining  the  time  needed 
to  complete  different  types  of  questions  most  commonly  used  in  Army  training  settings. 

Table  2 

Question  Formats  and  Estimated  Time  to  Complete 


Question  Format 

Time  per  Question 

Difficulty 

True-false 

20-30  seconds 

Easy 

Multiple-choice  (factual) 

40-60  seconds 

Moderate 

Multiple-choice  (complex) 

70-90  seconds 

Hard 

Matching  (5  stems/6  choices) 

2-4  minutes 

Variable 

Strictly  objective  questions,  such  as  true-false,  multiple-choice,  or  matching  tend  to  be 
the  most  useful  for  Army  training.  These  questions  are  relatively  quick  and  easy  to 
score  with  no  need  for  interpreting  answers.  Some  question  formats,  such  as  essay 
questions,  take  longer  to  answer,  tend  to  be  harder  to  grade,  and  increasingly  depend 
upon  some  degree  of  subjective  reasoning  on  the  part  of  the  grader  to  score.  While 
there  is  nothing  inherently  wrong  with  using  these  question  formats,  the  time  required 
for  answering  and  grading  them  and  the  need  to  standardize  subjective  judgments 
across  different  graders  often  makes  using  them  less  practical.  However,  this  guide 
assumes  that  you  will  most  likely  construct  questions  using  the  question  formats  listed 
in  Table  2  due  to  their  familiarity,  objectivity,  and  ease  of  scoring. 

Question  Difficulty  -  Paper-based 

You  must  also  consider  the  difficulty  of  the  questions.  Question  difficulty  in  this  guide  is 
described  as  easy,  moderate,  hard,  and  variable.  You  may  create  a  hard  knowledge- 
based  question  in  one  instance,  and  an  easy  application-based  question  in  another. 
There  is  variability  in  difficulty  with  each  question  format  depending  on  the  content  and 
how  the  question  is  worded.  However,  regardless  of  question  format,  in  general,  easier 
questions  require  less  time  to  answer  while  as  the  questions  increase  in  difficulty,  more 
time  is  needed.  The  example  questions  below  on  graphic  control  measures  illustrate 
different  degrees  of  difficulty. 

Easy  true-false  question: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “The  basic  symbol  for  a  friendly  ground  unit  is  a  rectangle.” 
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Moderate  (easy)  multiple  choice  question: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  following  is  the  correct  symbol  for  an  infantry  squad?” 


Moderate  (difficult)  multiple  choice  question: 

Circle  the  correct  letter  to  answer  the  following  question: 

“In  the  graphic  below,  at  which  location  would  you  place  a  FCL?” 


A.  At  location  “a” 

B.  At  location  “b” 

C.  At  location  “c” 

D.  At  location  “d” 


Hard  (complex)  multiple  choice  question: 

Circle  the  correct  letter  to  answer  the  following  question: 

“In  the  graphic  below,  select  the  proper  symbol  and  placement  for  a  FCL.” 


A.  “Solid  line”  at  location  “a” 

B.  “Dashed  line”  at  location  “a’ 

C.  “Solid  line”  at  location  “b” 

D.  “Dashed  line”  at  location  “b: 

E.  “Solid  line”  at  location  “c” 

F.  “Dashed  line”  at  location  “c” 

G.  “Solid  line”  at  location  “d” 

H.  “Dashed  line”  at  location  “d; 
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This  last  example  is  of  a  difficult  matching  question  that  requires  the  student  to  know 
multiple  facts  about  graphic  control  measures.  To  answer  all  portions  of  this  question 
correctly,  one  must  know  what  the  acronyms  stand  for  as  well  as  what  each  is  used  for. 
Also,  note  that  there  are  more  “functions”  than  “control  measures.”  This  helps  reduce 
the  impact  of  simple  guesswork  through  the  elimination  of  alternatives  as  known 
matches  are  selected. 


Difficult  matching  question: 

“Match  the  control  measure  in  the  first  column  with  its  function  in  the  second 
column  by  placing  a  letter  in  the  blank  next  to  a  number.” 


1.  FCL 

2.  LOA 

3.  FSCL 

4.  LD 

5.  FLOT 

6.  FEBA 

7.  LC 

8.  PL 

9.  FPF 

10.  RFL 


A.  Line  to  coordinate  departure  of  attack  elements 

B.  Designed  to  impede  enemy  movement 

C.  Used  to  lift  or  shift  fires 

D.  Used  for  coordination  of  air,  land,  and  sea  fires 

E.  Limits  of  ground  combat  units  except  screening/covering  units 

F.  Used  for  control  and  coordination  of  military  operations 

G.  Prohibits  fires  or  effects  from  fires  across  the  line 

H.  Forward  positions  of  friendly  forces 

I.  Boundary  of  area  in  which  gunfire  can  be  delivered 

J.  Where  two  opposing  forces  are  engaged 

K.  Terrain  feature  beyond  which  attacking  units  may  not  advance 


Cognitive  level  -  Paper-based 

You  may  also  need  to  determine  the  cognitive  level  of  each  question  prior  to  developing 
it.  The  cognitive  level  of  a  question  is  not  the  same  as  its  difficulty.  The  cognitive  level 
of  a  question  involves  the  first  three  levels  of  cognitive  understanding  as  commonly 
used  in  the  field  of  education.  These  levels  include  “factual”  knowledge  within  some 
particular  area,  “comprehension”  of  that  knowledge,  and  “application”  of  that  knowledge. 
Basic  knowledge  within  an  area  involves  remembering  facts,  terms,  concepts, 
definitions,  and  principles  that  pertain  to  the  subject  or  material  within  a  particular 
knowledge  area.  Comprehension  involves  a  further  understanding  of  those  facts  to 
include  what  they  mean  and  how  they  relate  to  other  facts.  Application  involves  an 
understanding  of  how  to  take  those  facts  and  make  them  useful  by  applying  them  to 
various  situations  to  solve  problems.  Each  of  the  four  levels  of  difficulty  may  be  applied 
in  each  of  the  three  levels  of  understanding. 


You  may  decide  to  formulate  questions  using  some  or  all  of  these  levels  depending  on 
what  you  deem  as  important  prior  knowledge.  Further  understanding  of  student 
knowledge  may  be  gained  with  questions  that  involve  comprehension  and  application 
rather  than  simple  fact-based  questions.  Comprehension  and  application  of  facts 
automatically  requires  knowledge  of  the  fact  itself.  You  will  have  to  determine  what 
levels  are  important  for  your  purposes.  As  an  example,  a  simple  fact  based  question 
may  be,  “From  the  pictures,  identify  the  screwdriver.”  A  comprehension  question 
involving  the  same  material  may  be,  “true  or  false:  Screwdrivers  are  used  to  tighten 
bolts.”  An  application  question  may  be,  “From  the  list  of  tools  to  the  right,  choose  the 
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proper  tool  to  remove  the  cover  on  the  piece  of  equipment  shown.”  All  three  questions 
involve  the  same  material  but  at  different  cognitive  levels.  You  must  decide  what 
cognitive  level(s)  you  will  use  to  design  the  questions  in  each  knowledge  area. 

To  help  with  construction  of  knowledge-based,  comprehension-based,  and  application- 
based  questions,  Table  3  provides  suggested  words  that  may  be  used  in  various  forms 
to  help  develop  questions  that  generally  relate  to  the  cognitive  level  of  the  question 
desired.  While  the  questions  do  not  necessarily  need  to  include  any  of  these  words, 
they  do  provide  a  basis  of  thought  during  question  development. 

Table  3 

Question  Cognitive  Level 1 


Asking  a: 

Involves: 

May  include  words: 

Knowledge  based 
question 

Remembering  facts,  terms, 
concepts,  definitions,  principles 

Define,  list,  state,  identify, 
label,  name,  who?,  when?, 
where?,  what? 

Comprehension 
based  question 

Understanding,  explaining,  or 
interpreting  the  meaning  of  material 

Explain,  predict,  interpret, 
infer,  summarize,  convert, 
translate,  give  example, 
account  for,  paraphrase 

Application  based 
question 

Using  a  concept  or  principle  in 
various  situations  to  solve 
problems 

Apply,  solve,  show,  make 
use  of,  modify,  demonstrate, 
compute 

As  an  example,  consider  the  following  three  simple  true-false  questions. 

T  F  “Lug  nuts  are  found  near  the  wheel  of  a  vehicle.” 

T  F  “To  remove  a  wheel  you  must  first  remove  the  lug  nuts.” 

T  F  “Lug  nuts  are  loosened  and  removed  with  a  counter  clockwise  action  using  the  lug 
wrench.” 

All  three  of  these  true-false  questions  are  asking  about  similar  content.  However,  the 
first  question  is  an  example  of  one  that  requires  the  student  to  recall  a  specific  fact.  In 
the  second  question  the  student  must  comprehend  the  function  of  lug  nuts  in  order  to 
answer  the  question  correctly.  In  the  third  question,  the  student  must  not  only 
understand  the  function  of  lug  nuts,  but  apply  that  knowledge  in  conjunction  with 
knowledge  and  understanding  of  tightening  and  loosening  lug  nuts  and  whether  the 
tools  listed  are  necessary  for  that  operation. 


1  Adapted  from  the  Indiana  University  resource  document  “How  To  Write  Better  Tests”, 
online  at  “http://www.indiana.edu/~best/pdf  docs/better  tests.pdf” 
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Formulating  Questions 

As  discussed  earlier  in  this  guide,  you  need  to  carefully  consider  the  content,  format, 
difficulty,  and  cognitive  level  of  each  question  in  your  assessment.  A  good  approach  is 
to  determine  what  basic  content  will  be  in  a  question  and  then  decide  what  format  would 
best  suit  that  content  and  your  desired  difficulty  and  cognitive  level.  For  example, 
matching  questions  work  well  with  knowledge  based  questions.  You  should  keep  in 
mind  your  purpose  behind  the  assessment  while  developing  your  questions.  Writing 
tricky  or  intentionally  misleading  or  trivial  questions  serves  little  purpose  in  determining 
whether  or  not  a  student  already  knows  or  comprehends  the  material  you  are  sampling. 
Trivial  information  that  does  not  adequately  assess  a  student’s  level  of  knowledge, 
comprehension,  or  understanding  of  key  ideas  and  concepts  will  not  assist  you  in 
determining  a  student’s  level  of  relevant  prior  knowledge. 

The  remainder  of  this  section  provides  some  considerations  and  tips  for  developing 
questions  in  different  formats2.  Question  examples  are  provided  to  illustrate  how  to 
develop  each  type  of  question.  A  variety  of  questions  used  in  different  actual  prior 
knowledge  assessments  is  provided  in  Appendix  D  as  additional  examples. 


Note:  The  following  questions  are  examples  that  demonstrate  different  question 
formats  and  to  show  how  questions  may  be  developed.  Exact  question  content  and 
wording  can  further  increase  the  difficulty  of  questions. 


True-false  Questions 

True-false  questions  are  generally  easy  to  construct  and  are  often  used  for  student 
recollection  of  factual  material  and  other  knowledge-based  information.  However,  they 
can  also  be  used  for  higher  level  cognitive  reasoning  requiring  comprehension  and 
application  of  knowledge.  As  shown  in  Table  2,  true-false  questions  take  relatively  little 
time  to  answer.  A  potential  drawback  to  the  use  of  true-false  questions  is  the  fact  that 
when  students  do  not  know  the  correct  answer,  they  have  a  50  /  50  chance  of  guessing 
right.  When  writing  a  true-false  question,  you  must  ensure  the  answer  is  unmistakably 
either  true  or  false. 


2  These  tips  were  adapted  from  the  Indiana  University  resource  document  “How  To 
Write  Better  Tests”,  online  at  “http://www.indiana.edu/~best/pdf  docs/better  tests.pdf” 
and  the  University  of  Waterloo  resource  document  “Exam  questions:  types, 
characteristics,  and  suggestions”,  online  at  “https://uwaterloo.ca/centre-for-teachinq- 
excellence/teachinq-resources/teachinq-tips/developinq-assiqnments/exams/questions- 

types-characteristics-suqqestions”. 
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Consider  the  following  guidelines  during  construction  of  true-false  questions. 


•  Ask  questions  involving  significant  or  major  content  and  avoid  trivial 
information. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “The  height  of  the  thread  on  a  course  thread  bolt  is  higher  than  the  thread 
on  a  fine  thread  bolt  of  the  same  size.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “To  tighten  most  common  bolts  you  would  turn  them  clockwise.” 

Comment:  Specific  knowledge  of  thread  height  on  bolts  can  be  considered  trivial 
and  non  essential  prior  knowledge  while  knowing  how  to  tighten  and  loosen  bolts 
is  a  common  mechanical  operation  that  is  more  relevant  to  vehicle  maintenance 
tasks. 


•  Make  sure  the  question  is  clearly  true  or  clearly  false. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “Antifreeze  is  used  to  fill  up  a  radiator.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “A  mixture  of  antifreeze  and  water  is  used  to  fill  up  a  radiator.” 

Comment:  Antifreeze  is  used  in  radiators  but  not  to  “fill  it  up.”  Water  is  also  used 
to  dilute  it.  Careful  construction  of  the  question  will  avoid  this  type  of  confusion. 

•  Avoid  trick  questions. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “A  tire  is  often  removed  using  a  lug  wrench.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “A  wheel  is  often  removed  using  a  lug  wrench.” 

Comment:  Wheels  are  commonly  removed  from  vehicles  by  using  a  lug  wrench 
to  remove  lug  nuts.  Tires  on  the  other  hand  are  mounted  to  the  wheels  and  are 
removed  using  various  other  tools  and  machines.  Trick  questions  may  assess 
the  attentiveness  of  a  student  but  not  necessarily  their  level  of  prior  knowledge. 
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•  Include  only  one  major  point  per  question. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “A  1/2  inch  six  point  socket  with  a  3/8  inch  drive  will  fit  a  1/2  inch  nut  as  long 
as  the  nut  does  not  screw  down  on  the  bolt  further  than  the  socket  can  reach  in 
which  case  a  1/2  inch  twelve  point,  deep  well  socket  or  a  1/2  inch  open  end 
wrench  may  be  used.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “A  1/2  inch  socket  with  a  3/8  inch  drive  will  fit  a  3/8  inch  bolt.” 

Comment:  More  than  one  main  point,  such  as  socket  points,  socket  drive,  and 
deep  well  sockets,  causes  confusion  and  can  become  an  exercise  in  deciphering 
the  question  rather  than  assessing  prior  knowledge.  Additionally,  longer  worded 
statements  may  often  be  answered  “False”  as  students  answering  may  believe 
that  some  portion  has  to  be  incorrect. 

•  Avoid  negatively  stated  questions. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “You  would  not  want  to  turn  a  bolt  clockwise  to  the  right  to  loosen  it.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “To  loosen  a  bolt  you  would  turn  it  counterclockwise,  to  the  left.” 

Comment:  Negatively  stated  questions  can  cause  a  student  to  answer 
incorrectly  even  when  the  correct  answer  is  known. 

•  Avoid  absolute  encompassing  words  such  as  “always”,  “all”,  or 
“never”. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “You  should  always  turn  bolts  to  the  right  to  tighten  them.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “You  should  generally  turn  bolts  to  the  right  to  tighten  them.” 

Comment:  Questions  including  all  encompassing  words  are  often  false  since 
relatively  few  instances  are  so  absolute.  As  a  general  rule,  most  bolts  are  turned 
to  the  right  to  tighten  them.  However  there  are  some  reverse  threaded  bolts  for 
specific  purposes  that  turn  to  the  left  to  tighten  them  rendering  the  answer  false  if 
the  term  “always”  is  used. 
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•  Avoid  long,  complex  questions. 

Poor: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “If  a  bolt  is  surrounded  by  other  material  and  difficult  to  reach  with  a  socket 
wrench,  and  you  have  tried  to  reach  it  with  an  open  end  wrench  but  failed,  and 
you  were  unable  to  remove  the  surrounding  equipment  to  better  reach  the  bolt,  a 
socket  wrench  extension  may  be  used  as  long  as  the  bolt  is  in  a  position  to 
where  the  extension  can  reach  it.” 

Better: 

In  response  to  the  following  statement  circle  T  for  true  or  F  for  false. 

T  F  “If  a  bolt  is  surrounded  by  other  material  and  difficult  to  reach  with  a  socket 
wrench,  a  socket  wrench  extension  may  be  of  use.” 

Comment:  Unnecessary,  wordy  information  that  does  not  add  relevant  content, 
meaning,  or  clarification  often  distracts  from  the  main  content.  Including  the 
failed  attempts  at  reaching  the  bolt  adds  little  to  the  information  necessary  for 
determining  whether  or  not  the  student  knows  the  purpose  of  a  socket  wrench 
extension. 

Multiple-Choice  Questions 

Multiple-choice  questions  are  one  of  the  most  common  formats  used  in  assessments 
due  to  their  objectivity  and  versatility.  Multiple-choice  questions  consist  of  two  parts 
including  a  stem  and  a  list  of  possible  solutions.  The  stem  of  the  question  presents  the 
first  part  of  a  problem  which  must  be  completed  or  answered  from  the  list  of  possible 
alternatives.  Stems  may  be  in  the  form  of  a  question  or  in  the  form  of  an  incomplete 
sentence  with  both  requiring  an  answer  from  the  accompanying  list  of  solutions.  In 
comparison  to  true-false  questions,  there  is  a  significant  reduction  in  the  ability  of  a 
student  to  correctly  guess  the  correct  answer  of  a  multiple-choice  question.  Unlike  true- 
false  questions  where  half  the  time  a  student  is  likely  to  guess  correctly,  multiple  choice 
questions  provide  greater  opportunities  to  reduce  the  impact  of  guessing  on  student 
scores.  Multiple-choice  questions  maintain  their  objectivity  by  providing  only  one 
correct  response  for  a  properly  worded  question.  In  addition,  they  provide  a  greater 
ability  to  assess  knowledge  and  understanding  across  all  three  cognitive  levels,  to 
determine  a  student’s  knowledge  of  facts  and  information,  to  measure  the 
comprehension  of  those  facts,  or  to  measure  subsequent  application  of  that  knowledge 
in  problem  solving.  While  not  necessarily  the  case,  multiple  choice  questions  that  tend 
to  be  more  complex  also  tend  to  assess  more  than  factual  knowledge.  As  shown  in 
Table  2,  complex  multiple-choice  questions  also  take  longer  to  answer  than  simpler 
questions.  While  multiple-choice  questions  can  be  difficult  and  time  consuming  to 
construct,  they  are  relatively  simple  and  easy  to  score.  During  development  of  multiple- 
choice  questions  you  should  review  the  considerations  given  for  true-false  questions  as 
well  as  the  following  general  guidelines. 
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•  Though  it  is  easier  to  write  low-level  knowledge  questions,  you 
should  also  include  questions  that  assess  higher  levels  of  cognitive 
understanding. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  following  fluids  goes  in  a  vehicle’s  cooling  system?” 

A.  Transmission  fluid 

B.  Antifreeze 

C.  Washer  fluid 

D.  Brake  fluid 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  following  fluids  is  commonly  required  to  repair  a  vehicle  whose 
engine  is  running  hot?” 

A.  Transmission  fluid 

B.  Antifreeze 

C.  Washer  fluid 

D.  Brake  fluid 

Comment:  More  information  can  be  determined  about  a  student’s  level  of 
understanding  when  questions  require  further  thought  rather  than  simple  recall. 
The  second  question  involves  both  diagnosis  and  solution  as  opposed  to  simple 
fact. 


•  Make  sure  the  stem  fully  states  the  problem  or  question. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  tools  is  best  to  use  on  a  wheel?” 

A.  Lug  wrench 

B.  Screwdriver 

C.  Hammer 

D.  Box  end  wrench 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  following  tools  is  best  suited  for  removing  the  nuts  on  a  wheel?” 

A.  Lug  wrench 

B.  Screwdriver 

C.  Hammer 

D.  Box  end  wrench 

Comment:  Leaving  room  for  interpretation  can  lead  to  confusion.  Without 
clearly  stating  what  is  required,  students  may  be  left  wondering  things  like  “Best 
in  what  sense?” 
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•  Place  words  in  the  stem  that  would  otherwise  be  repeated  in  the 
solution  options. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“A  socket  wrench  is:” 

A.  used  to  tighten  screws 

B.  used  to  remove  bolts 

C.  used  to  adjust  a  carburetor 

D.  used  to  install  wiper  blades 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“A  socket  wrench  is  used  to:” 

A.  tighten  screws 

B.  remove  bolts 

C.  adjust  a  carburetor 

D.  install  wiper  blades 

Comment:  Repeated  words  in  the  solution  distract  from  the  more  important 
information. 

•  Make  the  stem  as  concise  as  possible  without  losing  meaning. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“As  part  of  servicing  a  vehicle  for  routine  maintenance,  where  would  you  find  the 
engine  oil  drain  plug  to  drain  the  oil  in  order  for  you  to  replace  it  with  new  oil?” 

A.  Near  the  top  of  the  engine 

B.  On  the  bottom  of  the  transmission 

C.  On  the  bottom  of  the  oil  pan 

D.  Next  to  the  oil  cap 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Where  is  the  oil  drain  plug  located?” 

A.  Near  the  top  of  the  engine 

B.  On  the  bottom  of  the  transmission 

C.  On  the  bottom  of  the  oil  pan 

D.  Next  to  the  oil  cap 

Comment:  Unnecessary  or  excess  words  in  the  stem  that  do  not  add  relevant 
content,  meaning,  or  clarification  often  distract  from  the  more  important 
information.  The  fact  that  the  engine  is  being  serviced  or  that  new  oil  will  be  put 
into  the  engine  has  no  relevancy  to  the  student  knowing  where  the  oil  drain  plug 
is  located. 
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•  Ensure  there  is  only  one  correct  response  in  the  solution  list. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  following  fluids  is  commonly  required  to  repair  a  vehicle  engine  that 
is  running  hot?” 

A.  Transmission  fluid 

B.  Antifreeze 

C.  Water 

D.  Brake  fluid 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  following  fluids  is  commonly  required  to  repair  a  vehicle  engine  that 
is  running  hot?” 

A.  Transmission  fluid 

B.  Antifreeze 

C.  Washer  fluid 

D.  Brake  fluid 

Comment:  Having  more  than  one  correct  response  per  question  can  be 
confusing  even  if  the  student  knows  the  answer.  “Water”  and  “Antifreeze”  are 
both  coolants  for  the  engine. 

•  Provide  between  three  and  five  possible  solutions. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“A  1/2  inch  drive  socket  wrench  means:” 

A.  that  wrench  can  only  be  used  on  1/2  inch  bolts  and  nuts 

B.  the  connector  portion  of  the  wrench  to  the  sockets  is  1/2  inch 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“A  1/2  inch  drive  socket  wrench  means  the:” 

A.  wrench  can  only  be  used  on  1/2  inch  bolts  and  nuts 

B.  connector  portion  of  the  wrench  to  the  sockets  is  1/2  inch 

C.  wrench  can  only  be  used  on  nuts  and  bolts  of  1/2  inch  and  below 

D.  sockets  in  the  set  are  in  increments  of  1/2  inch 

Comment:  Too  few  solutions  increase  the  chance  that  students  can  guess  the 
correct  answer. 
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•  Solution  options  should  be  of  similar  nature  with  only  one  correct 
response. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Spark  plugs  are:” 

A.  usually  blue  in  color 

B.  made  completely  out  of  metal 

C.  used  to  ignite  gas  inside  a  gas  engine 

D.  not  used  in  diesel  engines 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Spark  plugs  are  used  to  ignite:” 

A.  diesel  inside  diesel  engines  only 

B.  gas  inside  diesel  and  gas  engines 

C.  gas  inside  a  gas  engine  only 

D.  gas  inside  a  gas  engine  and  diesel  inside  a  diesel  engine 

Comment:  The  intent  of  this  question  is  to  assess  whether  or  not  students  know 
that  spark  plugs  are  only  found  in  gas  engines  and  are  used  to  ignite  the  gas. 
Response  options  that  are  similar  in  nature,  such  as  in  the  second  question, 
require  the  student  to  think  more  deeply  in  order  to  decipher  which  is  the  correct 
answer. 

•  Avoid  using  “all  of  the  above”  or  “none  of  the  above”  as  these 
options  make  guessing  easier. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“In  the  list  below,  which  is  a  type  of  wrench?” 

A.  Socket 

B.  Box  end 

C.  Crescent 

D.  Lug 

E.  All  the  above 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“In  the  list  below,  which  is  a  type  of  wrench?” 

A.  Phillips 

B.  Vice  grip 

C.  Crescent 

D.  Needle  nose 

E.  Ball  peen 

Comment:  With  response  options  such  as  “All  the  above”,  students  who  know 
that  “crescent”  and  “socket”  are  both  a  type  of  wrench  do  not  have  to  know  that 
“box  end”  and  “lug”  are  also  in  order  to  guess  the  correct  “All  the  above” 
response. 
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•  Ensure  questions  are  stated  so  there  is  only  one  interpretation  of  the 
meaning  of  the  stem. 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“To  check  the  battery  on  a  vehicle  you  would:” 

A.  Attempt  to  crank  the  vehicle 

B.  Check  it  with  a  battery  analyzer 

C.  Attempt  to  make  the  terminal  spark  when  connecting  battery  cables 

D.  Look  at  the  battery  charging  indicator  inside  the  vehicle 

Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“To  check  the  voltage  and  cold  cranking  amps  of  a  vehicle  battery  you  would:” 

A.  Attempt  to  crank  the  vehicle 

B.  Check  it  with  a  battery  analyzer 

C.  Attempt  to  make  the  terminal  spark  when  connecting  battery  cables 

D.  Look  at  the  battery  charging  indicator  inside  the  vehicle 

Comment:  Checking  the  battery  could  mean  a  variety  of  things  such  as:  check  to 
see  if  it  is  there,  check  to  see  if  it  is  cracked,  check  to  see  if  it  will  crank  the 
vehicle,  check  to  see  what  the  current  cold  cranking  amps  are,  or  check  to  see  if 
it  is  hooked  up  properly. 

•  Consider  the  use  of  diagrams,  pictures,  charts,  tables  and  figures  for 
application  of  principles  and  concepts. 

Example  1 : 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“What  does  a  ball  peen  hammer  look  like?” 

A.  Flat  on  one  end  and  round  on  the  other 

B.  Flat  on  both  ends 

C.  Flat  on  one  end  with  a  claw  on  the  other 

D.  Flat  on  one  end  and  pointed  on  the  other 
Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Which  of  the  items  below  is  a  ball  peen  hammer?” 


C. 
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Example  2: 

Poor: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Where  are  the  spark  plugs  generally  located  on  a  gas  engine  vehicle?” 

A.  Near  the  cooling  fan 

B.  Near  exhaust  manifold 

C.  Near  the  distributer 

D.  Near  the  valve  covers 
Better: 

Circle  the  correct  letter  to  answer  the  following  question: 

“Where  are  the  spark  plugs  located  in  the  engine  diagram?” 

A.  1  _ 


Matching  Questions 

Matching  questions  consist  of  two  lists  in  which  individual  items  in  the  first  list  carry  a 
specific  association  with  an  item  from  the  second  list.  Items  within  both  lists  may  be  in  a 
variety  of  forms  to  include  sentences,  numbers,  single  words,  descriptions,  diagrams, 
and  labels.  Matching  questions  are  relatively  easy  to  create  for  factual  based 
knowledge  information  but  much  more  challenging  for  comprehension  and  application 
questions.  The  format  for  matching  questions  is  only  constrained  by  your  imagination 
as  you  develop  them.  However,  the  considerations  given  for  true-false  and  multiple- 
choice  questions  generally  apply  as  well  as  the  following  guidelines. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  28 


•  Always  include  more  responses  than  questions. 

Poor: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Hydraulic  fluid  for  the  master  cylinder 

4. 

Antifreeze 

D. 

Lubricant  for  the  engine 

5. 

Grease 

E. 

Lubricant  for  the  gears 

Better: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Comment:  Students  who  know  all  but  one  answer  will  correctly  answer  the  last 
item  if  only  one  choice  remains. 

•  Include  directions  that  clearly  state  the  basis  for  the  association 
between  the  items  in  the  two  lists. 

Poor: 

“Pick  the  correct  answers.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Better: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Comment:  Without  clearly  stating  what  is  required,  students  are  left  to  interpret 
what  they  believe  the  question  is  asking  them  to  do.  The  term  “correct”  may 
mean  different  things  to  different  students. 
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•  Items  in  each  list  should  be  of  similar  nature. 

Poor: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Tightens  and  loosens  nuts  and  bolts 

2. 

Lug  wrench 

B. 

Cleans  the  windshield 

3. 

Wiper  blades 

C. 

Test  the  cold  cranking  amps 

4. 

Battery  analyzer 

D. 

Used  to  remove  wheels 

5. 

Wrench 

E. 

Sets  gaps  in  spark  plugs 

F. 

Lubricant  for  the  engine 

Better: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Comment:  Listing  items  with  similar  characteristics  provides  a  better 
assessment  of  whether  or  not  the  student  understands  the  content  by  having 
them  decide  between  multiple  answers.  Items  with  different  characteristics  are 
often  easily  answered  due  to  elimination  of  answers  that  make  no  sense.  It 
makes  no  sense  that  wiper  blades  would  be  used  for  anything  in  the  list  other 
than  cleaning  the  windshield. 

•  Use  different  identification  systems  for  the  two  lists  (i.e. 
numbers/letters). 

Poor: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


A. 

Oil 

A. 

Coolant  for  the  engine 

B. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

C. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

D. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

E. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 
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Better: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 


placing  a  letter  in  the  blank  next  to  a 


1.  Oil 

A. 

2.  Brake  fluid 

B. 

3.  Transmission  fluid 

C. 

4.  Antifreeze 

D. 

5.  Grease 

E. 

F. 

number.” 

Coolant  for  the  engine 
Lubricant  for  the  ball  joints 
Lubricant  for  the  carburetor 
Hydraulic  fluid  for  the  master  cylinder 
Lubricant  for  the  engine 
Lubricant  for  the  gears 


Comment:  Using  letters  for  both  lists  could  be  confusing  to  the  student.  Using 
numbers  and  letters  helps  clearly  determine  which  two  items  match. 

•  Make  sure  all  responses  are  believable. 

Poor: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Fluid  for  waxing  a  vehicle 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Better: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Comment:  “Fluid  for  waxing  a  vehicle”  does  not  fit  into  the  same  common 
category  as  the  fluids  within  vehicle  components,  has  nothing  to  do  with 
mechanical  operations,  and  will  generally  be  ruled  out  as  a  believable  answer. 
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•  Make  sure  there  is  only  one  correct  response  per  question. 

Poor: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Water 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Fluid  for  the  radiator 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Better: 

“Match  the  item  in  the  first  column  with  its  function  in  the  second  column  by 
placing  a  letter  in  the  blank  next  to  a  number.” 


1. 

Oil 

A. 

Coolant  for  the  engine 

2. 

Brake  fluid 

B. 

Lubricant  for  the  ball  joints 

3. 

Transmission  fluid 

C. 

Lubricant  for  the  carburetor 

4. 

Antifreeze 

D. 

Hydraulic  fluid  for  the  master  cylinder 

5. 

Grease 

E. 

Lubricant  for  the  engine 

F. 

Lubricant  for  the  gears 

Comment:  Having  more  than  one  correct  response  per  question  can  be 
confusing  even  if  the  student  knows  the  answer.  “Water”  and  “Antifreeze”  are 
both  coolants  for  the  engine. 

Assessment  Length  -  Paper-based 

The  length  of  a  paper-based  assessment  depends  on  several  factors.  Some  of  these 
include  the  time  available  for  administration,  the  number  of  knowledge  areas  covered, 
the  number  of  questions  needed  to  adequately  sample  each  knowledge  area,  and  the 
question  format.  To  maximize  the  probability  that  the  assessment  will  provide  the  best 
estimate  of  relevant  prior  knowledge,  first  determine  the  question  content  and  the 
number  of  questions  in  each  knowledge  area  likely  to  meet  your  needs.  Then  estimate 
how  long  it  would  take  for  students  to  answer  those  questions.  If  only  a  fixed,  limited 
amount  of  time  is  allotted  for  the  administration  of  an  assessment,  then  the  maximum 
number  of  questions  can  be  determined  by  how  many  questions  in  the  desired  question 
formats,  students  will  likely  be  able  to  answer  during  that  time  period.  If  the  estimated 
time  to  answer  questions  is  longer  than  what  is  available,  consider  modifying  some  of 
the  question  formats  to  adjust  the  time  requirement,  or  do  a  quick  pilot  test  to  check  on 
the  time  to  complete  the  assessment. 

When  determining  the  number  of  questions  to  be  asked  in  each  knowledge  area  you 
should  consider:  the  amount  of  content  within  each  area  and  the  potential  impact  each 
area  has  on  student  performance  during  the  course  compared  to  other  knowledge 
areas.  Those  areas  with  more  content  and/or  more  potential  impact  would  likely  require 
more  questions  than  those  areas  with  less  content  and/or  less  potential  effect.  As  a 
general  rule,  you  may  start  with  a  basis  of  20  to  30  questions  per  area  and  adjust  up  or 
down  for  each  knowledge  area  as  needed. 
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Although  an  assessment  of  prior  knowledge  is  fundamentally  different  than  a  test  for 
knowledge  acquired  during  training,  the  principles  of  designing  either  a  test  or  an 
assessment  remain  basically  the  same.  The  number  of  questions  needed  to  determine 
some  level  of  understanding  is  similar  in  both  cases.  Research  has  shown  that  for  any 
particular  knowledge  area,  some  minimum  number  of  questions  is  needed  to 
adequately  sample  the  student’s  knowledge.  There  is  also  a  point  at  which  asking  more 
questions  becomes  futile  in  that  it  will  not  significantly  increase  the  accuracy  of  the  level 
of  knowledge  measured.  On  average,  the  number  of  questions  should  fall  somewhere 
between  20  and  30  for  any  given  knowledge  area.  In  other  words,  if  a  25-question 
assessment  containing  properly  designed  questions  revealed  a  score  of  75%  for  a  given 
student,  then  a  100-question  test  on  the  same  subject  matter,  at  the  same  level  of 
assessment,  would  not  significantly  alter  those  results  as  the  student  would  likely  score 
near  75%  on  that  test  as  well.  However,  asking  5  questions  on  the  same  subject  may 
not  be  enough  to  properly  assess  the  student’s  scope  and  depth  of  knowledge.  This 
generally  holds  true  for  any  particular  knowledge  area.  Assessments  that  include  more 
than  one  knowledge  area  will  need  to  include  an  adequate  number  of  questions  for 
each  area. 

Scoring  of  Assessments 

Consideration  should  also  be  given  as  to  how  students’  responses  will  be  scored.  Most 
scoring  methods  often  involve  assigning  one  point  to  each  question  and  giving  a  score 
based  on  a  raw  measure,  such  as  a  score  of  26  out  of  30  questions,  or  as  a  percentage 
such  as  87%,  derived  by  dividing  26  by  30.  Individual  scores  can  be  compared  to  other 
student  scores  to  give  an  indication  of  where  that  student  falls  within  the  group.  If  more 
than  one  knowledge  area  is  assessed,  consideration  should  be  given  as  to  if  and  how 
each  area  will  be  scored  independently.  Scoring  all  areas  as  a  single  group  may  or  may 
not  provide  you  the  information  you  need.  You  should  ask  yourself,  “Will  an  overall 
combined  score  of  all  areas  provide  the  information  needed  to  tailor  the  training 
envisioned  or  does  each  knowledge  area  need  to  be  scored  separately  to  appropriately 
tailor  training  in  each  area?” 
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ALC  Training  Practical  Example 

Step  5  -  Develop  the  questions  for  the  prior  knowledge  assessment 

Having  decided  on  what  content  will  be  covered  in  each  knowledge  area, 
SFC  Cooper  considers  the  number  and  type  of  questions  needed  for  each 
area.  He  decides  to  use  a  variety  of  question  types  in  his  assessment  that 
include  true-false,  multiple-choice,  and  matching  in  each  area.  Since  his 
objective  is  to  determine  which  students  do  not  have  a  basic  understanding  of 
the  selected  knowledge  areas,  he  decides  to  use  only  easy  or  moderately 
difficult  questions.  To  help  determine  students’  level  of  understanding,  he 
also  decides  to  ask  a  combination  of  factual,  comprehension,  and  application 
questions  in  each  knowledge  area. 

Since  the  instructor  views  all  three  knowledge  areas  essential  to  helping 
him  tailor  training  appropriately,  SFC  Cooper  decides  that  due  to  the  amount 
of  content  in  graphic  symbols  and  operations  orders  that  he  will  include  more 
questions  in  each  of  these  areas.  He  finally  decides  to  ask  4  true-false  and 
multiple-choice  questions  on  map  reading  and  8  true-false  and  multiple-choice 
questions  each  on  graphic  symbols  and  operations  orders  for  a  total  of  20 
questions.  He  also  plans  on  developing  one  matching  question,  consisting  of 
5  items,  for  each  knowledge  area  to  produce  an  assessment  of  35  possible 
correct  answers.  He  decides  that  scoring  one  point  for  each  correct  answer 
will  provide  the  best  way  to  compare  individual  student  results.  Since  the 
tailored  training  plans  will  not  change  based  on  the  specific  results  from  each 
knowledge  area,  he  decides  to  use  a  combined  score  across  all  knowledge 
areas.  Based  on  the  number  and  types  of  items  in  his  assessment,  SFC 
Cooper  plans  to  allow  30  minutes  for  students  to  complete  the  assessment 
(see  Table  2). 
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Chapter  3.  Validating  Prior 
Knowledge  Assessments 

3.1  What  is  Validation? 

In  this  guide,  validation  means  ensuring  your  prior  knowledge  assessment  detects  the 
students  you  designed  it  to  identify.  In  simple  terms,  validation  basically  means 
confirming  the  assessment  meets  your  needs.  There  are  several  of  types  of  validity. 

For  the  purposes  of  this  guide,  validation  of  prior  knowledge  assessments  will  center  on 
the  use  of  predictive  validity.  Predictive  validity  is  the  extent  to  which  scores  on  an 
assessment  predict  scores  on  subsequent  performance. 

As  such,  your  assessment  is  only  as  good  as  its  use  in  predicting  performance  outcome 
to  an  acceptable  degree.  The  closer  the  assessment  comes  to  predicting  actual 
performance,  or  being  related  to  actual  performance,  the  better  tool  it  is.  In 
mathematical  terms,  the  strength  of  this  relationship  is  reflected  in  the  correlation 
between  the  two  measures.  Validation  is  the  process  of  checking  to  ensure  that  an 
acceptable  relationship  exists  between  an  assessment  and  actual  performance. 
Validation  should  be  completed  prior  to  using  an  assessment  to  predict  subsequent 
training  outcomes.  Once  an  assessment  is  validated,  it  may  be  used  for  multiple 
iterations  of  training  unless  revalidation,  as  explained  in  Chapter  4,  Using  and  Revising 
Prior  Knowledge  Assessments,  is  necessary. 

A  key  and  necessary  assumption  is  made  in  regards  to  the  prior  knowledge  assessment 
you  have  designed.  It  is  assumed  that  the  assessment  scores  for  a  given  group  of 
students  during  the  validation  phase  are  representative  of  scores  that  will  be  generated 
by  students  taking  the  same  prior  knowledge  assessment  in  subsequent  classes.  This 
assumption  is  necessary  in  order  to  make  informed  decisions  about  tailored  training 
using  the  information  gained  from  assessment  results. 

When  validating  your  assessment,  you  will  compare  a  measurement  before  existing 
training  with  a  measurement  after  existing  training.  These  two  measurements  are 
considered  to  be  related  if  their  outcomes  are  associated  in  a  consistent  manner.  In 
other  words,  if  the  assessment  measurement  systematically  correspond  to  actual 
performance  measurements,  meaning  a  student’s  score  on  one  indicates  a  score  on  the 
other,  then  they  are  considered  to  be  related.  As  an  example,  a  glass  jar  containing 
some  volume  of  sand  weighs  a  certain  amount.  Adding  more  sand  to  the  jar  increases 
the  weight  and  the  volume.  Adding  even  more  sand  correspondingly  increases  the 
weight  and  volume  once  again.  Therefore,  the  first  measurement,  the  volume  of  sand  in 
the  glass,  is  related  to  the  second  measurement,  the  weight  of  the  sand  in  the  jar.  This 
type  of  relationship  is  directly  proportional  and  very  consistent  since  the  volume  of  sand 
directly  relates  to  the  weight. 
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However,  measurements  can  often  be  described  as  being  related  even  though  their 
relationship  is  less  consistent  than  increasing  volume  and  weight.  For  example,  the 
more  horsepower  a  car  has,  the  faster  it  can  generally  go.  The  relationship  is  not 
always  consistent  however.  Other  factors  such  as  weight  and  aerodynamic  design  can 
also  influence  speed.  But  there  still  remains  a  relationship  between  horsepower  and 
speed  which  allows  for  some  general  conclusions  about  speed  given  horsepower.  A 
poor  relationship  would  exist  where  one  measure  has  little  or  no  consistent  relationship 
with  the  second  measure.  An  example  of  a  poor  relationship  would  be  the  color  of  a  car 
and  how  fast  it  can  go,  since  color  has  virtually  no  effect  on  car  speed. 

The  usefulness  of  a  prior  knowledge  assessment  can  be  quantified  by  the  strength  of 
the  relationship  between  the  scores  on  the  assessment  and  scores  on  the  criterion 
performance  measure.  The  strength  of  this  relationship  can  be  stated  in  terms  of  a 
correlation  coefficient  (described  later  in  steps  3  and  4  of  the  validation  procedure, 
paragraph  3.4  of  this  guide).  If  predictions  based  on  a  prior  knowledge  assessment  fail 
to  reflect  how  students  actually  perform  after  training,  the  assessment  is  of  little  use. 
Keep  in  mind,  assessments  will  not  be  able  to  offer  predictions  with  100%  accuracy. 
Validation  comes  from  obtaining  a  correlation  coefficient  with  a  value  high  enough  to 
support  use  of  the  assessment.  To  assist  in  validation,  this  chapter  includes 
instructions  for  developing  an  Excel  spreadsheet  to  compute  values  of  a  correlation 
coefficient. 

3.2  What  are  the  Input  Values  Used  for  Validation? 

There  are  three  sets  of  input  required  for  validation.  They  are  the  scores  on  the 
assessment,  the  scores  on  the  criterion,  and  student  identification  information.  The 
assessment  scores  are  obtained  when  the  prior  knowledge  assessment  is  administered. 
The  criterion  scores  are  obtained  at  the  end  of  some  specified  training  which  is  not  yet 
tailored,  and  therefore  represent  student  performance  typical  of  non-tailored  training. 

For  purposes  of  validation,  assessment  scores  are  not  used  to  tailor  scheduled 
training  in  any  way,  but  rather  to  support  the  assessment’s  subsequent  use. 

Drawing  any  conclusions  from  assessment  results  prior  to  validation  can  lead  to 
incorrect  conclusions  regarding  which  students  should  be  identified  for  tailored  training. 
Validation  is  necessary  in  order  to  determine  if  there  is  a  relationship  between  selected 
prior  knowledge  and  actual  student  performance.  Altering  training  based  on  pre¬ 
validation  assessment  results  at  this  point  would  alter  student  subsequent  performance 
and  leave  no  way  of  determining  whether  the  results  predicted  by  the  prior  knowledge 
assessment  were  meaningful  or  accurate. 

3.3  How  Do  I  Establish  My  Performance  Criterion  for 
Validation? 

The  criterion  you  use  for  validation  is  a  performance  measurement  based  on  the 
objectives  of  the  course  or  block  of  training  in  which  tailored  training  is  planned.  The 
key  to  establishing  the  criterion  is  to  first  ensure  it  is  based  on  course  or  training  block 
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objectives.  Likely  this  measurement  exists  in  some  form  already  within  the  course  or 
can  be  derived  from  available  test  scores,  evaluation  scores,  or  course  performance 
ratings.  However,  criterion  measurements  should  be  obtained  from  measurements 
potentially  impacted  by  the  presence  of  relevant  prior  knowledge,  or  lack  thereof.  Using 
criterion  measurements  based  on  ratings  not  associated  with  relevant  prior  knowledge 
can  lead  to  misguided  decisions  on  which  students  may  benefit  from  tailoring.  While 
course  performance  measurements  may  already  exist,  they  may  also  include  ratings 
based  on  performance  not  impacted  by  the  assessed  prior  knowledge.  Criterion 
measurements  should  not  include  measurements  not  relevant  or  potentially  relevant  to 
the  presence  of  prior  knowledge. 

Criterion  measurements  may  be  obtained  at  the  completion  of  all  training  or  at  some 
point  earlier.  A  determination  of  when  to  collect  criterion  measurements  will  have  to  be 
made  depending  on  what  is  being  measured  and  how  the  performance  it  measures 
relates  to  other  training  within  the  course.  Learning  in  courses  that  are  sequential  in 
nature  tends  to  build  upon  concepts  and  information  as  the  course  progresses.  Since 
early  learning  in  the  course  impacts  later  learning,  criterion  measurements  may  best  be 
obtained  after  initial  training  of  the  identified  knowledge  area(s).  You  should  keep  in 
mind  that  collecting  criterion  measurements  early  in  training  often  offers  data  better 
representative  of  the  effects  of  prior  knowledge,  as  these  effects  tend  to  fade  as  training 
progresses. 

In  the  sequential  progressive  training  for  the  course  shown  in  Figure  1  below,  Test  1 
measures  performance  on  training  provided  up  to  that  point.  Performance  on  Tests  2, 

3,  and  4  build  upon  earlier  learning.  In  this  case  it  may  be  best  to  establish  the  criterion 
at  point  A  based  on  the  results  of  Test  1  rather  than  at  point  B  following  the  completion 
of  Test  4.  In  that  way  performance  results  early  in  the  course  can  be  compared  to 
assessment  results  to  determine  the  impact  of  relevant  prior  knowledge  on  early 
learning. 


Start  of  Course 


Complete  Course 
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A 
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HI 
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Assessment 


Figure  1.  Course  X  -  sequential  progressive  training. 

Other  courses  are  structured  with  various  blocks  of  instruction  that  are  not  dependent 
on  learning  between  blocks.  In  these  courses,  performance  in  one  block  may  have  little 
impact  on  performance  in  subsequent  blocks.  Tailoring  in  these  courses  may  be  limited 
to  certain  blocks  of  instruction. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  37 


In  the  course  shown  in  Figure  2  below,  training  is  broken  into  four  separate  training 
blocks.  Each  block  is  independent  of  the  others  and  each  block  test  only  covers 
material  contained  within  that  block.  In  this  case,  the  criterion  may  best  be  established 
at  the  end  of  the  training  block  in  which  tailoring  will  be  conducted.  In  this  example, 
tailoring  is  envisioned  for  block  3  and  the  criterion  based  on  its  associated  test  at  point 
A. 
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Complete  Course 


Block  4 
Test 


Figure  2.  Course  Y  -  independent  blocks  of  training. 

A  prior  knowledge  assessment  is  only  as  good  as  its  ability  to  provide  an  indication  of 
how  students  are  likely  to  perform  on  the  criterion  measure.  As  such,  assessments 
used  to  predict  performance  must  first  be  validated  to  determine  if  there  is  a  relationship 
between  performance  in  the  assessment  and  on  the  criterion  measure  you  identified. 
This  is  the  heart  of  the  validation  process.  In  simple  terms,  a  validated  assessment  is 
one  where  the  presence  of  prior  knowledge,  as  indicated  by  the  assessment  scores, 
provides  an  indication  of  what  the  related  criterion  measure  score  would  be.  Thus, 
criterion  measurements  should  include  measurements  only  related  or  potentially 
impacted  by  the  presence  of  prior  knowledge. 

Training  institutions  have  various  ways  in  which  students  are  evaluated  on  their  course 
performance  while  in  training.  Some  of  those  evaluations  include  performance 
measurements  which  may  not  be  affected  by  the  prior  knowledge  areas  covered  in  the 
assessment.  For  example,  many  courses  include  student  performance  on  a  physical 
fitness  test  in  the  final  course  grade.  Since  physical  ability  is  most  likely  not  linked  to 
the  prior  knowledge,  the  criterion  selected  for  comparison  should  not  include  the 
physical  fitness  test  performance. 

To  continue  the  example  of  maintenance  training  for  tracked  vehicle  mechanics 
described  in  Chapter  2,  step  3  of  assessment  development,  recall  that  the  instructor 
decided  to  assess  student  prior  knowledge  of  tools  and  electrical  wiring.  In  preparation 
for  validation,  suppose  the  assessment  was  given  at  the  beginning  of  training  and  each 
student  received  an  assessment  score.  It  is  now  the  end  of  training.  To  validate  the 
assessment,  student  performance  must  now  be  measured  in  order  to  make  a 
comparison.  In  this  particular  training  course  the  students  were  given  an  overall  rating 
based  on  the  combined  scores  of  written  tests,  hands-on  tests,  Army  Physical  Fitness 
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Test  (APFT)  scores,  and  an  instructor  subjective  participation  score.  Should  this  overall 
rating  of  combined  scores  be  used  for  comparison?  The  answer  is  “no”.  Prior 
knowledge  of  tools  and  electrical  wiring  has  no  effect  on  how  well  students  score  on  the 
AFPT  and  little  to  do  with  how  well  they  participate  in  training.  The  assessed  prior 
knowledge  will  be  related  to  at  least  some  of  the  written  or  hands-on  tests.  Therefore, 
what  performance  measurement  (criterion)  should  be  used?  In  this  case,  only  the 
written  or  hands-on  tests  that  are  directly  maintenance  related  should  be  used.  An 
appropriate  criterion  may  be  the  combination  of  all  maintenance  related  test  scores 
including  written  and  hands-on  tests. 

If  an  assessment  related  criterion  is  not  available  as  a  matter  of  practice  for  any 
particular  training,  you  will  have  to  create  a  measurement  in  order  to  validate  the 
assessment.  Possibilities  include  separating  out  test  scores  or  other  measurements  for 
material  impacted  by  prior  knowledge  from  those  that  do  not  apply  or  by  introducing  a 
post-training  measurement  or  knowledge  check  that  currently  does  not  exist.  In  any 
case,  in  order  for  the  assessment  to  be  worthwhile,  the  criterion  must  be  based  on 
course  objectives  and  requirements  potentially  impacted  by  the  knowledge  areas 
covered  in  the  prior  knowledge  assessment. 


You  should  also  determine  not  only  what  the  criterion  will  be  but  also  how  it  will  be 
quantified.  Measurements  identified  with  a  number  such  as  a  percentage  or  a  score 
tend  to  be  more  useful  for  comparisons  than  other  less  quantifiable  measurements  such 
as  in  a  dichotomous  “Go/No-Go”  or  “Pass/Fail”  system.  Dichotomous  measurements 
simply  indicate  whether  or  not  a  student  met  the  objective,  but  they  give  no  indication  of 
how  well  or  how  poor  the  objective  was  met.  Measurements  that  give  an  indication  of 
the  degree  of  how  good  or  how  bad  a  student  performs  are  more  useful  for  comparison 
purposes  than  simple  Go/No-Go  measurements. 


Dichotomous  Criterion  Measurements 

There  may  be  some  training  courses  or  instances  where  dichotomous  measurements 
are  the  only  available  or  reasonable  alternative  for  establishment  of  a  criterion.  You 
should  keep  in  mind  that  there  may  be  creative  ways  to  turn  simple  measurements  into 
more  quantifiable  ones.  For  example,  two  students  passed  two  hands-on  tests  during  a 
block  of  training.  Student  #1  passed  the  first  test  on  his  first  try,  but  it  took  him  two  tries 
to  pass  the  second.  Student  #2  took  three  tries  to  pass  the  first  test  and  four  tries  to 
pass  the  second.  A  simple  calculation  using  this  information  could  be  performed  to 
provide  a  more  complete  assessment  of  students’  performance. 

Student  #1  Student  #2 

Number  of  tests  2  2 

Total  number  of  attempts  3  7 

Overall  percent  correct  (2/3  =  67%)  (2/7  =  29%) 


While  this  score  has  no  effect  on  whether  the  students  pass  the  training  or  not,  it  does 
give  a  better  indication  of  each  student’s  actual  performance.  Consideration  should 
also  be  given  to  whether  overall  “Go/No-Go”  or  “Pass/Fail”  scores  can  be  evaluated  in 
more  detail  to  give  a  more  precise  indication  of  performance.  For  example,  suppose  a 
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student  receives  a  “No-Go”  on  a  particular  test  because  the  standard  was  not  met. 
Further  examination  may  reveal  that  the  test  was  made  up  of  20  distinct  steps. 

Although  the  student  correctly  performed  19  of  the  20  steps,  one  critical  step  was 
performed  incorrectly  and  an  overall  “No-Go”  was  given.  Compare  this  to  a  student  that 
receives  an  overall  “No-Go”  but  performed  15  of  the  20  steps  incorrectly  or  a  student 
who  received  a  “Go”  with  5  incorrect  steps.  Using  data  regarding  the  completion  of 
steps  or  number  of  errors  committed  to  compile  a  continuous  numerical  score,  even 
within  different  categories  of  errors  (i.e.  procedural  or  critical)  if  the  steps  are  weighted 
differently  in  the  evaluation,  could  provide  more  sensitive  data  and  further  distinguish 
between  actual  performance  of  individual  students. 

Dichotomous  measurements  of  final  attempts  fail  to  recognize  occasions  such  as  when 
a  student  struggles  with  the  training  during  instruction  but  eventually  passes  after 
repeated  tries.  When  using  dichotomous  measurements,  you  must  use  scores  that  best 
distinguish  between  student  performances.  For  example,  if  you  decide  to  use  test 
results  from  the  first  hands-on  test  within  a  course  for  your  criterion,  then  as  a  general 
rule  of  thumb  you  would  need  to  use  the  first  attempt  test  results  for  that  test  as  your 
criterion  measurement.  In  courses  where  students  are  retrained  and  retested  until 
virtually  all  “Go’s”  are  achieved  by  all  students,  then  using  the  final  results  would  not 
distinguish  one  student  from  another  as  all  students  would  eventually  have  the  same 
score.  Using  first  attempt  results  as  a  criterion  better  distinguishes  between  student 
performances  and  provides  a  more  sound  foundation  for  validating  your  prior  knowledge 
assessment. 

3.4  What  is  the  Validation  Procedure? 

To  validate  an  assessment  you  must  conduct  a  statistical  analysis  to  determine  the 
relationship  between  the  scores  from  the  prior  knowledge  assessment  and  the  criterion 
measurement  obtained  after  training.  This  requires  some  mathematical  calculations 
which  can  be  accomplished  using  an  Excel  spreadsheet.  Before  creating  a 
spreadsheet,  you  should  ensure  you  have  all  of  the  required  data. 

Table  4  below  provides  steps  to  be  used  to  validate  an  assessment  for  subsequent  use 
in  tailoring. 

Table  4 

Prior  Knowledge  Assessment  Validation  Steps 


Validation  Steps 

Step  1 

Administer  the  Prior  Knowledge  Assessment 

•  Administer  the  assessment 

•  Instructor  does  not  see  the  results 

•  Instructor  conducts  training  as  usual 

•  Collect  criterion  scores  after  training 
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Validation  Steps 

Step  2 

Obtain  and  Input  data  into  an  Excel  spreadsheet 

•  Obtain  student  names  and/or  roster  numbers,  prior  knowledge 
assessment  scores,  criterion  measurement  scores 

•  Create  an  Excel  spreadsheet 

•  Record  three  sets  of  data  in  the  spreadsheet  (name/roster 
number,  assessment  score,  criterion  score) 

•  Check  all  data  for  accuracy 

Step  3 

Compute  a  correlation  coefficient 

•  Create  correlation  coefficient  formula  in  the  spreadsheet 

•  Compute  the  correlation  coefficient 

Step  4 

Evaluate  the  strength  of  the  relationship  between  the  two  scores 
•  Determine  if  the  relationship  is  strong  enough  to  support  using 
the  prior  knowledge  assessment  to  tailor  training 

Step  5 

Plot  the  relationship  between  the  two  scores 

•  Create  scatter  plot  in  Excel  using  input  data 

•  Examine  the  scatter  plot  for  outliers  and  strength  of  the 
relationship 

Step  1  -  Administer  the  Prior  Knowledge  Assessment 

To  validate  your  assessment  you  must  administer  it  to  a  group  of  students  in  the  same 
manner  as  when  using  it  for  tailoring  training.  It  should  be  administered  within  the  first 
few  days  of  training  since  the  more  students  are  exposed  to  training  content  the  more 
influence  their  training  can  have  on  their  results.  The  objective  is  to  obtain  an 
understanding  of  what  prior  knowledge  each  student  enters  the  classroom  with  prior  to 
any  instruction,  familiarization  with  training  content,  or  knowledge  shared  by  more 
confident,  knowledgeable  students.  Consequently,  the  earlier  the  assessment  can  be 
administered  the  better. 

In  order  to  validate  a  prior  knowledge  assessment,  you  must  have  a  sufficient  number  of 
students  for  training  that  will  produce  both  prior  knowledge  assessment  scores  and 
criterion  scores.  Generally,  as  the  number  of  students  used  in  the  validation  process 
increases,  the  more  reliable  the  conclusions  are  about  the  relationship  of  prior 
knowledge  assessment  scores  to  criterion  scores.  As  a  general  rule,  you  should  have 
at  least  30  students  who  will  be  given  the  prior  knowledge  assessment  and  complete 
the  training  in  order  to  validate  the  assessment.  If  the  class  size  is  less  than  30,  you 
may  need  to  administer  the  prior  knowledge  assessment  to  more  than  one  class  and 
obtain  criterion  scores  from  all  students. 

Administer  the  assessment  in  the  same  way  you  would  administer  any  closed-book, 
individual  effort  test  or  exam.  Be  careful  not  to  downplay  the  importance  of  the 
assessment  by  overemphasizing  that  it  will  not  be  a  part  of  the  course  grade  as  this 
may  lead  to  students  not  performing  to  their  full  ability.  Once  the  students  have 
answered  all  the  questions,  collect  the  completed  assessments. 

The  next  action  is  extremely  important.  Once  prior  knowledge  assessments  are 
collected,  they  must  be  stored  for  use  at  the  end  of  training.  The  instructor 
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conducting  the  training  should  not  score  or  view  the  results  of  the  assessment 

during  the  validation  process.  If  needed,  they  can  be  administered  and/or 
stored/scored  by  you  the  developer  or  another  training  administrator  or  supervisor  and 
the  results  secured  appropriately.  The  key  point  is  for  instructors  not  to  view  the  results 
until  the  end  of  training  to  ensure  they  do  not  inadvertently  allow  that  knowledge  to  alter 
training  during  the  validation  trial.  Ultimately,  the  assessment  results  will  be  used  to 
enhance  student  performance;  however,  not  before  the  assessment  instrument  has 
been  validated.  The  training  for  this  block  or  course  should  be  conducted  in  exactly  the 
same  manner  as  usual,  as  if  the  initial  assessment  had  never  been  administered. 

At  the  completion  of  the  training  block  or  course,  you  are  ready  to  validate  your 
assessment.  You  will  now  need  to  compute,  collect,  and  record  the  criterion  results 
according  to  your  plan  for  obtaining  those  scores.  After  the  criterion  measure  has  been 
administered,  the  prior  knowledge  assessment  can  now  be  scored  as  well.  Thus,  each 
student  should  have  a  prior  knowledge  assessment  score  and  a  performance  based 
criterion  score.  Student  results  that  do  not  contain  both  sets  of  scores,  for  any  reason, 
must  not  be  used  in  validation.  You  now  have  all  of  the  information  you  need  to  validate 
the  assessment:  students’  identity  or  name,  prior  knowledge  assessment  scores,  and 
criterion  scores. 


Note:  While  there  are  a  great  number  of  different  statistical  programs,  e.g.  SPSS, 
that  could  be  used  during  this  process,  they  require  special  permissions  to  load  on  a 
government  computer,  additional  resources  to  purchase,  and  additional  training  in 
their  operation.  Since  Excel  is  part  of  the  Windows  Office  suite  loaded  onto  nearly 
every  Army  computer,  it’s  readily  available  to  all  personnel  and  nearly  everyone  has 
at  least  some  familiarity  with  its  functions. 


Step  2  -  Obtain  and  Input  Data  Into  an  Excel  Spreadsheet 

After  obtaining  prior  knowledge  assessment  and  criterion  scores  from  all  of  the 
students,  you  are  ready  to  input  the  data  into  an  Excel  spreadsheet.  The  following 
procedures  will  explain  each  step.  Appendix  C  contains  sample  exercises  for  the  Excel 
procedures  explained  below. 

Using  the  procedures  below  carefully  input  and  double  check  all  student  data. 
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1 .  Open  MS  Excel  to  a  blank  worksheet. 
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3.  Identify  students  by  roster  number  and/or  name. 
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4.  Input  assessment  and  criterion  scores  for  each  student.  Double  check  your  input 
data  for  errors.  It  may  be  worthwhile  to  have  a  separate  individual  check  all  the 
data  after  it  has  been  entered.  Be  sure  that  both  scores  are  matched  to  the 
correct  student.  The  example  below  shows  criterion  scores  based  on  continuous 
numerical  measurements  exhibiting  a  range  of  scores. 
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5.  If  your  criterion  measurement  is  in  the  form  of  dichotomous  measurements  (i.e, 
measures  with  only  possible  outcomes)  your  criterion  scores,  such  as  “Go  /  No- 
Go”  or  “Pass  / Fail”,  will  have  to  be  converted  into  a  number  in  order  to  enter  them 
into  Excel  for  further  computations.  In  order  to  do  this  you  will  need  to  convert  all 
of  one  type  score  into  “1  ’s”  and  all  of  the  other  type  into  “0’s”.  “1  ’s”  normally  are 
associated  with  “Go”  or  “Pass”  and  “0’s”  with  “No-Go”  or  “Fail”.  In  this  case  all 
“Go”  or  “Pass”  scores  would  be  converted  to  a  “1”  and  all  “No-Go”  or  “Fail” 
scores  would  be  converted  to  a  “0”.  Use  these  numbers  for  criterion  scores  for 
each  student.  The  example  below  shows  how  those  scores  would  be  entered. 
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Note:  The  procedures  in  Steps  3  through  5  are  the  same  for  data  containing  both 
continuous  numerical  and  dichotomous  measurements. 


Step  3  -  Compute  the  Relationship  Using  Excel 

After  entering  the  data  you  will  need  to  enter  the  proper  formula  into  Excel  in  order  to 
compute  a  relationship  in  the  form  of  a  correlation  coefficient.  A  correlation  coefficient  is 
easily  calculated  using  Excel.  Use  the  following  procedures  to  compute  a  correlation 
coefficient  from  the  data  you  entered.  The  example  below  shows  continuous  numerical 
measurement  data. 

1 .  Position  the  cursor  in  cell  “E2”  underneath  “Correlation”  and  click  the  right  mouse 
button.  This  will  bring  up  a  menu.  With  the  left  mouse  button  click  on  “Format 
Cells”. 
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2.  This  will  bring  up  a  new  menu.  Make  sure  the  “Number”  tab  is  selected  from  the 
tabs  at  the  top  of  the  menu  and  with  the  left  mouse  button  click  on  “Number” 
underneath  “Category:”  Ensure  “Decimal  places:”  are  set  at  “2”  and  with  the 
mouse  button  left  click  on  “OK”  at  the  bottom  of  the  screen. 
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3.  Position  the  cursor  back  in  cell  “E2”  underneath  “Correlation”  and  click  the  left 
mouse  button  ensuring  cell  “E2”  is  highlighted  with  a  bold  box. 


4.  Type  the  following:  =CORREL( 

As  you  type  you  will  see  “CORREL(array  1, array  2)”  appear  with  “array  1”  in 
bold  print  near  cell  “E2”. 
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5.  With  “array  1”  in  bold  print,  move  your  cursor  and  click  on  the  first  number  in 
column  “C”  with  the  left  mouse  button  and  without  releasing  the  mouse  button 
drag  the  cursor  to  the  last  number  in  column  “C”.  Now  release  the  mouse  button 
and  this  will  draw  a  box  around  the  assessment  scores  in  column  “C”  only.  Make 
sure  you  only  include  the  scores  in  column  “C”  with  no  extra  spaces  at  the  end 
and  do  not  include  the  title.  As  you  drag  the  cursor  you  will  see  the  cell  positions 
for  “array  1”  automatically  record  in  your  function. 
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6.  Type  a  comma  ( ,  )  and  “array2”  will  automatically  show  in  bold  type. 
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7.  With  “array  2”  in  bold  print,  take  your  cursor  and  click  on  the  first  number  in 
column  “D”  with  the  left  mouse  button  and  without  releasing  the  mouse  button 
drag  the  cursor  to  the  last  number  in  column  “D”.  Now  release  the  mouse  button 
and  this  will  draw  a  box  around  the  assessment  scores  in  column  “D”  only.  Make 
sure  you  only  include  the  scores  in  column  “D”  with  no  extra  spaces  at  the  end 
and  do  not  include  the  title.  As  you  drag  the  cursor  you  will  see  the  cell  positions 
for  “array  2”  automatically  record  in  your  function. 
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8.  Hit  the  “Enter”  key  on  your  keyboard  and  the  correlation  for  your  assessment  will 
automatically  be  calculated  and  displayed  in  the  cell  in  which  you  typed  the 
function.  The  final  formula  should  appear  as  shown. 


9.  If  your  data  includes  dichotomous  measurements  the  correlation  coefficient 
formula  is  entered  in  the  same  manner  as  shown  in  the  example  below. 
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10.  Observe  the  correlation  coefficient  and  once  again  check  for  any  errors  during 
input  of  data  or  functions.  If  you  receive  an  error  message  while  entering  the 
function,  delete  what  you  have  entered  and  attempt  to  reenter  it  using  the  steps 
above.  If  you  are  not  familiar  with  Excel  and  are  having  difficulty  you  may  need 
to  ask  for  assistance  from  someone  more  familiar  with  the  program.  Once  you 
have  obtained  your  correlation  coefficient  proceed  to  Step  4  for  evaluation  of  the 
value  you  obtained. 


Note:  Additional  instructions  in  Appendix  B  include  procedures  for  adding,  changing, 
and  deleting  input  data. 


Step  4  -  Evaluate  the  Strength  of  the  Relationship  Between  the  Two  Scores 

The  correlation  coefficient  you  calculate  between  the  prior  knowledge  assessment  score 
and  the  criterion  score  reflects  the  relationship  between  the  two  scores  and  will  range 
between  -1 .0  and  +  1 .0.  So  what  does  this  mean? 

•  Positive  correlation  coefficients  mean  that  high  scores  on  one  measure  are 
associated  with  high  scores  on  the  other,  and  low  scores  on  one  are  associated 
with  low  scores  on  the  other.  Positive  values  between  0.0  and  1 .0  indicate  a 
positive  relationship. 

•  Negative  correlation  coefficients  mean  that  high  scores  on  one  measure  are 
associated  with  low  scores  on  the  other,  and  that  low  scores  on  one  are 
associated  with  high  scores  on  the  other.  Negative  values  between  -1 .0  and  0.0 
indicate  a  negative  relationship  between  the  measurements. 

•  A  correlation  coefficient  of  0.0  means  that  scores  on  one  measure  do  not  indicate 
the  magnitude  of  scores  on  the  other  and  there  is  no  association  between  the 
two  scores.  Values  at  or  near  0.0  indicates  there  is  no  relationship  between  the 
measurements. 


Although  both  positive  and  negative  relationships  are  possible,  in  general,  for  your 
purposes,  the  correlations  you  will  obtain  will  fall  between  0  and  +1.  In  other  words,  you 
will  probably  have  instruments  where  high  assessment  scores  reflect  good  performance 
and  high  criterion  scores  reflect  good  performance  as  well.  The  closer  the  correlation  is 
to  +1 ,  the  stronger  the  relationship.  This  indicates  less  error  in  prediction  from  your 
prior  knowledge  assessment  to  the  criterion  measure.  For  example,  if  instructors  are 
interested  in  providing  additional  assistance  to  students  likely  to  struggle  with  the 
training,  a  strong  relationship  indicates  less  error  involved  in  identifying  early  on  the 
students  requiring  that  assistance. 
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After  determining  the  correlation  coefficient,  you  need  to  evaluate  the  value  obtained  to 
determine  if  the  strength  of  the  relationship  between  a  student’s  prior  knowledge  and 
subsequent  performance  is  strong  enough  to  validate  the  assessment’s  usefulness. 
Higher  correlation  values  indicate  better  relationships  and  more  confidence  in 
forecasting  outcomes  while  values  close  to  0.0  indicate  less  of  a  relationship  and  less 
confidence. 

Generally  correlation  coefficients  that  involve  performance  scores  fall  somewhere 
between  0.3  and  0.5  with  correlations  between  0.5  and  1 .0  possible  but  seen  much  less 
often.  Values  less  than  0.3  are  generally  considered  weak  relationships  and  the 
assessment’s  usefulness  as  a  predictor  is  suspect. 

As  a  rule  of  thumb  to  guide  you,  once  you  have  obtained  a  correlation  coefficient  of  0.3 
or  above,  your  prior  knowledge  assessment  provides  a  sufficiently  strong  relationship  to 
the  criterion  to  make  decisions  regarding  individual  differences  and  tailored  training. 
However,  higher  values  of  the  correlation  will  enable  instructors  to  make  more  sound 
decisions  regarding  individuals.  Once  you  have  validated  your  assessment,  it  may  be 
used  for  subsequent  tailored  training  to  target  students  the  assessment  was  designed  to 
identify. 

If  the  correlation  coefficient  falls  below  the  acceptable  limit,  the  assessment  will  most 
likely  need  to  be  altered.  Consideration  should  then  be  given  to  a  reanalysis  of  either 
the  knowledge  areas  chosen  for  the  assessment  or  of  the  questions  asked  within  each 
area.  See  Chapter  4  of  this  guide  for  further  guidance  on  assessments  that  are  not 
validated. 

Student  scores  can  also  be  reflected  on  a  graph  for  a  clearer  view  of  the  relationship.  A 
perfect  relationship  means  that  if  you  know  the  value  of  one  score  you  can  accurately 
predict  the  value  of  the  other  score  and  are  indicated  with  correlation  coefficients  at  the 
extremes  of  +1 .0  and  -1 .0.  Strong  relationships  are  represented  by  numbers  near  -1 .0 
or  1 .0.  Weak  relationships  are  represented  by  numbers  near  0.0. 

The  graphs  in  Figures  3,  4,  and  5  below  illustrate  three  correlations  involving  continuous 
numerical  measurements  (a  positive  correlation  at  or  near  +1 .0,  a  negative  correlation 
at  or  near  -1 .0,  and  no  correlation  at  0). 
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Figure  3.  Positive  correlation  with  continuous  numerical  measurements. 
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Figure  4.  Negative  correlation  with  continuous  numerical  measurements. 
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No  Correlation 
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Figure  5.  No  correlation  with  continuous  numerical  measurements. 


The  graphs  in  Figures  6,  7,  and  8  below  illustrate  three  correlations  involving 
dichotomous  criterion  measurements  (a  positive  correlation  at  or  near  +1 .0,  a  negative 
correlation  at  or  near  -1 .0,  and  no  correlation  at  0). 
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Figure  6.  Positive  Correlation  with  dichotomous  criterion  measurements. 
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Negative  Correlation 
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Figure  7.  Negative  correlation  with  dichotomous  criterion  measurements. 


No  Correlation 


•  Correlation 
Coefficient 
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Figure  8.  No  correlation  with  dichotomous  criterion  measurements. 
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Step  5  -  Plot  the  Relationship  Between  the  Two  Scores  -  Examine  for  Outliers 

To  help  visually  observe  and  further  interpret  your  data,  a  scatter  plot  may  be  used. 
Scores  for  each  student  can  be  represented  by  a  data  point  on  a  scatter  plot  graph  with 
two  axes  where  one  axis  represents  the  assessment  score,  and  the  other  axis  the 
criterion  measurement.  Each  individual  student’s  scores  can  be  plotted  on  this  graph  by 
the  representation  of  one  data  point.  When  all  student  data  points  are  plotted,  they 
generally  form  some  type  of  cluster  of  data  points  whether  that  be  small,  large,  circular, 
linear,  or  some  other  type  of  pattern. 

For  your  purposes  during  validation,  visualization  of  scores  on  a  scatter  plot  can  help 
you  look  for  outliers.  An  outlier  is  a  data  point  that  for  some  reason  does  not  fall  within 
the  reasonable  limits  or  general  cluster  of  the  rest  of  the  data  points.  Outliers  can  be 
caused  by  a  wide  variety  of  factors  including,  but  not  limited  to:  incorrect  data  entries, 
students  outside  the  general  population,  incorrect  data  used  for  input  entries,  or  external 
factors  affecting  the  results  of  input  data.  Outliers  can  skew  your  analysis  and  provide 
results  that  may  not  truly  represent  the  relationship  of  your  measurements.  For  each 
outlier,  a  determination  must  be  made  of  whether  its  associated  data  should  be 
justifiably  excluded.  Once  a  scatter  plot  is  produced,  you  should  visually  look  for 
outliers.  Consider  the  example  in  Figure  9. 


Student  Scores 


60  65  70  75  80  85  90  95  100 


Prior  Knowledge  Assessment  Score 
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Figure  9.  Relationship  of  student  scores  (continuous  numerical  criterion  scores). 

You  will  notice  that  the  data  points  are  plotted  from  data  including  continuous  numerical 
criterion  scores  and  generally  cluster  along  a  linear  path  sloping  up  and  to  the  right  with 
a  correlation  coefficient  of  0.47.  However,  you  should  also  note  one  data  point  falls  well 
outside  the  cluster.  This  point  would  be  considered  an  outlier.  This  is  important 
because  this  inconsistency  is  likely  due  to  some  undetermined  reason  that  may  cause 
you  to  exclude  that  data  point  from  your  analysis.  Inconsistent  data  points  such  as  the 
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one  above  can  cause  inappropriate  variations  in  the  analysis.  Once  you  have  identified 
any  possible  outlier,  you  should  then  further  analyze  that  particular  point  for  contributing 
factors  that  would  cause  the  outlier,  other  than  what  was  measured  by  the 
assessments,  and  subsequently  decide  if  that  point  should  be  excluded  from  the 
analysis. 

For  the  outlier  shown  in  Figure  9,  you  would  normally  expect  that  either  the  criterion 
measurement  would  be  higher  (between  the  mid  80’s  and  mid  90’s)  or  the  assessment 
score  would  be  lower  (around  mid  70’s  or  below)  in  order  to  cluster  with  the  other  data 
points.  A  further  analysis  of  that  data  point  may  reveal,  as  one  possibility,  that  an  input 
error  was  made.  It  could  also  reveal  some  other  contributing  factor.  For  example,  if 
cumulative  test  scores  were  used  as  the  criterion  and  a  student  was  absent  on  some 
test  days,  the  criterion  score  would  be  inaccurate.  The  missing  scores  would  cause  a 
low  criterion  measurement  but  would  not  necessarily  be  indicative  of  how  that  student 
would  have  performed  had  the  tests  been  taken.  This  would  be  a  logical  reason  for 
excluding  that  data  point  from  the  analysis.  You  may  also  consider  excluding  from  the 
analysis  those  students  who  are  not  part  of  what  would  be  considered  the  general 
population  and  may  likely  skew  the  data,  e.g.  allied  students  or  students  from  other 
services  who  would  be  expected  to  lack  the  prior  knowledge  you  are  measuring. 

You  can  also  visually  observe  for  outliers  in  scatter  plot  graphs  that  include 
dichotomous  criterion  measurements.  Consider  the  graph  in  Figure  10  below. 
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Figure  10.  Relationship  of  student  scores  (dichotomous  criterion  scores). 

You  will  notice  that  data  points  in  Figure  10  generally  cluster  in  one  of  two  groups. 
Students  that  generally  score  lower  on  the  assessment  also  tend  to  score  lower  on  the 

GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  60 


criterion.  Students  that  generally  score  higher  on  the  assessment  also  tend  to  score 
higher  on  the  criterion.  Outliers  in  this  case  would  be  seen  as  a  data  point  falling 
outside  one  of  these  clusters.  In  Figure  10  one  of  data  points  is  clearly  outside  of  either 
cluster  of  points  and  may  indicate  an  issue  with  that  particular  data. 

When  dealing  with  potential  outliers,  you  must  be  careful  not  to  exclude  applicable  data 
points  for  the  sake  of  intentionally  trying  to  alter  your  analysis  in  order  to  validate  the 
assessment.  Remember,  the  reason  validation  is  necessary  is  to  establish  that  there  is 
a  credible  relationship  between  your  measurements. 

Tweaking  input  data  or  excluding  data  points  to  intentionally  alter  your  analysis 
can  lead  to  faulty  conclusions  which  could  include  targeting  the  wrong  students 
for  the  tailored  training  you  envision. 

To  create  your  own  scatter  plot  using  the  data  you  entered  into  an  Excel  spreadsheet, 
use  the  following  procedures. 
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1 .  To  use  the  scatter  plot  function,  draw  a  box  around  both  sets  of  data  by  placing 
the  cursor  in  the  first  cell  of  the  assessment  scores,  cell  “C2”,  clicking  the  left 
mouse  button,  and  without  releasing  the  left  mouse  button  drag  the  cursor  to  the 
last  number  of  the  criterion  scores,  in  this  example  cell  “D32”,  and  releasing  the 
left  mouse  button.  This  will  draw  a  box  around  all  the  numerical  scores  you  have 
entered. 
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2.  Once  both  sets  of  data  are  highlighted,  click  on  the  “Insert”  tab  near  the  top  of  the 
page  to  display  the  insert  options. 
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3.  Move  the  cursor  over  the  scatter  plot  option,  left  click  with  the  mouse,  and  an 
option  box  will  appear.  Move  the  cursor  down  to  the  first  option  that  shows  data 
points  without  lines  labeled  “Scatter  with  only  markers”. 


Note:  When  a  scatter  plot  is  generated  in  Excel,  the  first  column  of  data,  in  this  case 
“Assessment  Score”,  will  be  the  “X  axis”  and  the  second  column  of  data,  “Criterion 
Score”,  will  be  the  “Y axis.” 
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4.  Left  click  on  the  “Scatter  with  only  markers”  option  and  a  scatter  plot  will 
automatically  be  generated.  The  scale  along  the  bottom  of  the  chart  reflects 
“Assessment  Scores”  and  the  scale  up  the  left  side  of  the  chart  reflects  “Criterion 
Scores”.  The  scatter  plot  generated  may  be  used  to  help  identify  outliers.  The 
scatter  plot  below  shows  a  possible  outlier  and  a  correlation  coefficient  of  0.19. 
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5.  If  you  determine  that  an  outlier  exists  recheck  your  input  data.  If  an  error  exists 
correct  it  and  the  correlation  coefficient  and  scatter  plot  will  be  updated 
automatically.  If  there  is  sufficient  justification  for  eliminating  the  outlier,  simply 
delete  all  information/data  on  the  student.  Procedures  for  correcting  input  data 
are  outlined  in  Appendix  B  of  this  guide.  After  correcting  the  input  error,  the 
correlation  coefficient  is  adjusted  to  0.34. 
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6.  Follow  the  same  procedure  for  creating  a  scatter  plot  that  includes  dichotomous 
data. 
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Chapter  4.  Using  and  Revising  Prior 
Knowledge  Assessments 

4.1  How  Do  I  Determine  Which  Students  Will  Receive 
Tailored  Training? 

After  a  prior  knowledge  assessment  has  been  validated,  subsequent  assessment 
results  can  be  used  to  determine  which  students  would  most  benefit  from  tailored 
training.  A  student’s  actual  assessment  score  probably  has  less  meaning  in  isolation 
than  when  compared  to  other  students’  scores.  For  example,  a  score  of  46  on  a  70 
point  assessment  has  little  meaning  when  viewed  in  isolation.  When  compared  to  30 
other  students  with  all  of  them  scoring  below  46,  you  quickly  see  a  different  story.  The 
opposite  could  also  be  true.  If  the  majority  of  other  students  scored  above  46,  this 
student  could  be  someone  destined  to  struggle  during  training  without  additional 
support. 

Once  you  have  all  the  scores  in  hand,  you  must  determine  where  to  make  the  defining 
cut  between  those  likely  to  need  assistance  and/or  those  needing  to  be  challenged  due 
to  their  higher  level  of  prior  knowledge.  There  is  no  standard  rule  for  establishing  these 
cut  points  and  deciding  where  to  draw  the  line  can  be  a  challenging  task.  For  example, 
an  arbitrary  decision  of  the  lower  1/3  of  the  class  may  not  accurately  capture  all  the 
students  needing  tailored  training.  Like  many  other  tests,  these  decisions  often  become 
easier  only  after  repeated  experience. 

Initial  prior  knowledge  assessment  scores  used  during  validation  can  identify  what  you 
should  likely  expect  from  a  typical  group  of  students.  Examining  the  results  from  your 
validation  can  help  determine  starting  cut  points  for  students  who  should  receive 
tailored  training  in  subsequent  classes.  Carefully  considering  the  data,  both  in  raw  form 
and  on  a  scatter  plot,  can  provide  you  an  idea  of  where  students  will  generally  score  on 
the  assessment  and  how  those  scores  generally  relate  to  subsequent  training 
performance. 

Some  factors  that  can  impact  your  decision  of  where  to  establish  a  cut  point  include: 

•  Past  student  performance  on  the  criterion  measure.  -  Experience  with 
previous  student  performance  in  the  same  training  course  on  the  criterion 
measure  can  help  you  distinguish  between  high  and  low  performers.  In  the 
validation  process,  you  will  therefore  be  looking  at  scores  on  the  prior  knowledge 
assessment  that  correspond  to  individuals  performing  at  these  high/low  levels  on 
the  criterion  measure. 

Example:  Results  from  previous  courses  indicate  that  students  who 
tended  to  have  little  trouble  advancing  through  training  often  scored  above 
93%  on  the  criterion  measure.  They  were  high  performers  and  individuals 
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who  you  think  should  be  challenged  if  you  were  to  tailor  the  course  to 
address  their  needs.  Similarly,  students  who  struggled  tended  to  score 
65%  and  below  on  the  same  criterion.  These  students  performed  poorly 
and  you  believe  they  would  have  benefited  from  tailored  training  that 
addressed  their  problems.  The  next  step  is  to  determine  cut  points  on  the 
prior  knowledge  test  which  best  correspond  to  these  levels  of  performance 
on  the  criterion. 

•  Visual  analysis  of  results  -  Are  there  any  natural  groupings  or  clear  indications 
of  where  cut  points  on  the  prior  knowledge  assessment  might  be  established  by 
considering  either  the  validation  scatter  plot  or  prior  knowledge  assessment 
results?  Initially  scatter  plots  of  these  results  may  provide  some  insights  into 
how  students  are  grouped  on  the  assessment. 

Example:  Visually  considering  the  validation  scatter  plot  revealed  that 
students  who  scored  above  90%  on  the  prior  knowledge  assessment  also 
seemed  to  perform  extremely  well  on  the  criterion  measure. 

•  Strength  of  the  validation  correlation  coefficient  -  Does  the  strength  of  the 
correlation  coefficient  potentially  have  any  effect  on  deciding  where  to  establish 
the  cut  point?  The  stronger  the  relationship  between  assessment  results  and 
demonstrated  performance,  the  more  confident  one  can  be  in  predicting  how 
students  will  perform.  Establishment  of  cut  points  in  weaker  relationships  means 
that  some  students  selected  for  tailoring  may  not  need  it  and/or  some  students 
not  chosen  may  have  benefited  more  had  they  received  it. 

Example:  The  validation  results  showed  a  correlation  coefficient  of  0.33 
between  the  assessment  scores  and  the  criterion.  While  this  correlation 
coefficient  can  be  used  to  help  establish  cut  points,  it  will  likely  ensure  that 
some  lower  performers  on  the  criterion  may  be  included  with  higher 
scorers.  The  instructor  decides  to  err  on  the  side  of  caution  and  provides 
more  students  with  additional  assistance.  He  also  plans  to  observe 
students  during  training  for  signs  of  those  that  could  be  removed  or  added 
to  that  training. 

•  Prior  experience  with  a  Prior  Knowledge  Assessment  -  Has  the  prior 
knowledge  assessment  been  used  in  previous  training?  Previous  use  of  prior 
knowledge  assessments  is  one  of  the  best  sources  of  information  for 
establishing  cut  points.  An  examination  of  student  assessment  results  coupled 
with  the  cut  point  used  for  that  training  and  the  associated  student  subsequent 
performance  results  can  help  determine  whether  the  cut  point  established  should 
be  adjusted  or  not. 

Example:  Prior  experience  in  using  the  assessment  shows  that  generally 
students  who  score  below  75%  on  the  assessment  could  benefit  from 
additional  assistance. 
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•  Resources  available  -  Do  resources  potentially  have  an  effect  on  how  many 
students  will  receive  tailoring?  Resource  impacts  may  include  a  limitation  of 
equipment  and/or  facilities  used  in  tailoring,  a  limitation  of  instructor  personnel 
needed  to  administer  tailoring,  and/or  a  limitation  of  the  time  available  to  conduct 
tailoring. 

Example:  Available  resources  may  limit  tailoring  to  only  the  bottom  or  top 
10  students  in  the  class. 

Consider  the  scatter  plot  examples  in  Figures  11,  12,  and  13  generated  from  validation 
results. 
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Figure  11.  Relationship  of  student  scores  (0.53  correlation  coefficient). 
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Figure  13.  Relationship  of  student  scores  (0.31  correlation  coefficient). 

In  the  figures,  assume  that  resources  are  not  a  limitation  and  that  the  instructor  has  no 
prior  experience  with  the  prior  knowledge  assessment. 

Identifying  high  performers  -  those  who  need  to  be  challenged.  Assume  you 
know  that  high  performers  on  the  criterion  test  score  at  least  90%.  Consider  the  vertical 
line  at  point  “A”.  With  stronger  correlation  coefficients  as  in  Figures  1 1  and  12  there  is  a 
natural  grouping  of  four  students  who  scored  above  80%  on  the  prior  knowledge 
assessment  and  also  performed  well  in  the  course  as  evidenced  by  a  criterion  score  of 
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88%  or  above.  Although  you  assume  that  90%  and  above  is  the  desired  level  on  the 
criterion,  only  one  of  the  4  students  scored  below  90%  with  this  individual  still  having  a 
relatively  high  score  on  the  criterion  (88%).  Assuming  similar  results  in  the  future,  an 
80%  cut  point  on  the  prior  knowledge  test  seems  appropriate.  Note,  however,  that  there 
were  other  individuals  who  scored  90%  and  above  but  had  prior  knowledge  scores 
below  80%.  Thus  the  prior  knowledge  assessment  did  not  uniquely  identify  all  the  high 
performers  on  the  criterion.  With  the  weaker  correlation  in  Figure  13,  natural  groupings 
of  students  are  not  as  apparent  as  in  Figures  1 1  and  12.  In  Figure  13  with  a  weaker 
correlation  coefficient,  the  grouping  is  not  as  tight  as  evidenced  by  the  wider  dispersion 
of  criterion  scores,  with  two  students  who  scored  above  80%  on  the  prior  knowledge 
assessment,  but  below  90%  on  the  criterion.  If  you  used  the  80%  prior  knowledge  cut 
point  based  on  these  data,  you  might  find  that  challenging  some  of  these  students  is  not 
successful  and  they  should  receive  the  “regular”  instruction. 

Now  consider  the  vertical  line  “B”  of  73%  on  the  prior  knowledge  test.  In  all  three 
figures,  this  cut  point  is  not  acceptable  for  identifying  high  performers  -  those  who  need 
to  be  challenged.  In  each  case  the  spread  of  criterion  scores  has  increased  and 
includes  many  individuals  who  did  not  score  above  90%  on  the  criterion. 

Identifying  low  performers  -  those  who  need  assistance.  Now  assume  that  a 
criterion  score  of  below  80%  is  associated  with  low  performers.  With  the  stronger 
correlations  shown  in  Figures  1 1  and  12  you  will  note  that  there  are  a  number  of 
students  to  the  left  of  line  “B”  who  did  not  score  80%  or  better  on  the  criterion  as  well  as 
a  number  of  students  who  did.  However,  you  will  notice  at  vertical  line  “C”  that  three  out 
of  the  four  students  to  the  left  of  line  “C”  scored  below  80%  on  the  criterion  with  the 
fourth  fairly  close  at  83%.  You  will  also  notice  that  all  four  of  these  students  scored 
below  63%  on  the  prior  knowledge  assessment.  Making  the  assumption  that  this 
association  holds  true  in  the  future,  using  63%  as  a  prior  knowledge  score  cut  point  for 
students  requiring  tailoring  seems  reasonable.  Therefore,  line  “C”  would  be  a  better  cut 
point  than  line  “B”.  In  the  weaker  correlation  example  shown  in  Figure  13,  the  grouping 
of  the  scores  to  the  left  of  line  “C”  was  more  widely  dispersed  as  students’  prior 
knowledge  assessment  scores  ranged  from  some  near  90%  to  others  near  70%.  Again, 
a  lower  correlation  makes  it  more  challenging  to  determine  cut  points. 

These  examples  simply  give  an  idea  of  how  you  may  use  the  validation  scatter  plot  data 
to  help  visualize  groupings  of  students  and  help  establish  cut  points.  Sorting  of  prior 
knowledge  assessment  results  in  ascending  or  descending  order  can  also  help  visualize 
groupings  of  students  and  aid  in  the  establishment  of  cut  points.  Sorting  of  assessment 
results  can  be  accomplished  easily  in  Excel.  An  example  of  sorting  of  data  in  Excel  is 
provided  in  Appendix  B. 

You  should  remember  that  the  relationship  between  the  prior  knowledge  measured  on 
the  assessment  and  actual  student  performance  in  training  will  not  be  perfect.  In 
essence  this  means  that  due  to  the  lack  of  a  perfect  association,  some  students 
assumed  to  excel  in  the  course  may  not  and  some  students  assumed  to  have  trouble  in 
the  course  may  excel.  This  in  turn  may  exclude  some  students  from  tailored  training 
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who  need  it  and  include  some  students  who  do  not  depending  on  where  you  ultimately 
decide  to  establish  your  cut  points  for  students  to  participate  in  tailored  training. 


4.2  How  Do  I  Administer  a  Prior  Knowledge  Assessment? 

Administration  of  a  prior  knowledge  assessment  is  accomplished  in  the  same  manner 
as  any  other  test  in  a  classroom.  As  discussed  previously,  the  assessment  should  be 
given  within  the  first  few  days  of  arriving  for  training.  Whether  the  assessment  is 
administered  for  the  first  time  for  validation  or  subsequently  for  identifying  students  for 
tailored  training  makes  no  difference  in  administration  procedures. 

Administration  Procedures 

During  administration  for  validation,  assessments  should  be  distributed  to  all  students 
who  will  potentially  be  included  in  the  validation  process.  Students  should  be  given  an 
adequate  amount  of  time  to  complete  the  assessment.  Remember,  the  intent  behind 
the  prior  knowledge  assessment  is  to  determine  the  students’  extent  of  prior  knowledge, 
not  their  reading  speed  or  comprehension.  Therefore,  be  sure  to  consider  the  reading 
and  English  language  capability  of  the  students  when  establishing  the  time  allowed  for 
completing  the  assessment. 

In  order  to  obtain  the  most  valuable  data,  students  should  be  encouraged  to  perform  to 
their  best  ability  on  the  assessment.  Students  who  feel  the  assessment  will  not  impact 
their  grade  or  progress  in  training  may  be  inclined  not  to  answer  questions  to  their  best 
ability.  Ensure  the  attitude  displayed  to  students  during  assessment  administration 
encourages  their  best  performance. 

Collecting  the  Results 

If  being  used  for  validation,  collect  assessment  results  and  secure  them  for  future  use. 
At  the  appropriate  time,  you  will  also  need  to  collect  criterion  results  for  validation 
purposes.  Validated  assessments  used  for  tailoring  training  are  scored  immediately 
after  administration  to  determine  which  students  should  be  targeted  for  tailored  training. 

After  collecting  results  from  an  assessment  to  be  used  for  tailoring,  you  should  use 
whatever  conditions  were  set  to  identify  the  students  for  tailored  training.  Conditions 
may  either  be  based  on  a  set  number  of  individuals,  a  set  assessment  score,  a 
combination  of  both,  or  others  you  determine.  For  example,  you  may  want  to  target 
those  students  scoring  below  a  certain  assessment  score  but  may  only  be  able  to 
accommodate  10  students  for  tailoring.  This  could  mean  you  have  less  than  ten 
students  identified  for  tailoring  if  the  class  scored  well  or  you  may  have  to  select  the  10 
lowest  scores  if  more  than  10  students  scored  below  your  target  cutoff  score.  Once 
targeted  students  have  been  identified,  you  have  accomplished  the  purpose  behind  the 
prior  knowledge  assessment. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  72 


4.3  What  if  the  Assessment  was  Not  Valid? 


After  developing  a  prior  knowledge  assessment  and  performing  all  steps  for  validation, 
you  will  have  calculated  a  correlation  coefficient.  As  discussed  previously,  a  validated 
assessment  will  generally  have  a  correlation  coefficient  greater  than  0.3.  Assessments 
that  did  not  validate  have  correlation  coefficients  less  than  0.3.  Since  use  of  these 
assessments  can  lead  to  misleading  conclusions  about  which  students  should  be 
targeted  for  tailoring,  you  would  not  be  able  to  use  an  assessment  with  a  correlation 
coefficient  of  less  than  0.3  in  its  current  form.  All  may  not  be  lost  however.  Your 
assessment  may  still  be  of  use.  An  analysis  of  the  correlation  coefficient  will  help  you 
determine  if  modifying  the  assessment  would  likely  improve  the  likelihood  of  validation. 

Correlation  Coefficient  Analysis 

The  first  thing  you  should  consider  for  an  assessment  that  did  not  validate  is  how  close 
it  came  to  validating.  One  would  agree  that  the  difference  between  a  correlation 
coefficient  of  0.29  and  0.3  is  very  slight  while  the  difference  between  correlation 
coefficients  of  0.1  and  0.3  is  much  greater.  While  not  strong  enough  to  support  using 
the  assessment  to  predict  later  performance,  correlation  coefficients  that  approached 
0.3  can  indicate  that  some  relationship  exists  and  modifications  may  make  it  a  useful 
assessment.  On  the  other  hand,  weaker  correlation  coefficients  or  correlation 
coefficients  that  are  closer  to  0.0  indicate  the  assessment  will  either  require  major 
modifications  or  may  not  be  useful  at  all.  Looking  at  the  main  causes  for  an 
assessment  not  validating  will  help  determine  if  modifying  your  existing  prior  knowledge 
assessment,  or  criterion  measure,  is  more  prudent  than  developing  a  new  assessment. 

Potential  Causes  for  Not  Validating 

There  are  a  variety  of  possibilities  for  why  prior  knowledge  assessments  fail  validation, 
all  of  which  cannot  be  covered  here.  However,  there  are  a  few  major  factors  to  consider 
that  may  aid  in  understanding  the  cause  of  the  assessment  not  validating.  Reviewing 
these  factors  can  assist  in  revising  or  redesigning  your  prior  knowledge  assessment  to 
obtain  a  more  predictive  tool.  Major  factors  include: 

•  Areas  Assessed  -  The  prior  knowledge  areas  assessed  may  have  little  or  no 

bearing  on  actual  student  performance. 

•  Questions  Used  -  The  questions  developed  may  not  properly  sample  the  level 

of  prior  knowledge  in  the  areas  assessed. 

•  Criterion  Used  -The  criterion  used  may  include  measurements  of  student 

performance  outside  those  potentially  impacted  by  the  presence  of  prior 
knowledge. 

•  Timing  of  Criterion  Measurement  -  The  measurement  used  to  develop  a 

criterion  score  was  taken  at  the  wrong  point  in  training. 

Areas  Assessed 

For  correlation  coefficients  that  indicate  weak  relationships,  you  and  the  instructors 
should  review  your  basic  assumptions  about  the  relationship  of  prior  knowledge  to 
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student  performance.  Prior  to  developing  the  assessment,  you  made  an  assumption 
that  having  prior  knowledge  in  certain  knowledge  areas  would  assist  students  in  their 
performance  throughout  training  in  that  course  or  block  of  instruction.  It  is  worth 
revisiting  that  assumption  to  scrutinize  whether  or  not  you  still  believe  it  is  valid.  It  may 
be  that  prior  knowledge  in  those  areas  has  little  to  do  with  whether  or  not  the  student 
can  or  will  perform  well  in  training.  If  you  and  the  instructors  determine  that  to  be  the 
case,  then  a  reanalysis  of  what  prior  knowledge  areas  may  potentially  impact 
performance  should  be  accomplished.  There  may  be  knowledge  areas  that  were  not 
assessed  that  may  be  better  related  to  performance  outcomes.  It  may  also  hold  true 
that  prior  knowledge  in  any  particular  area  may  not  help  predict  student  performance  for 
that  specific  training. 

If  you  and  the  instructors  determine  that  other  knowledge  areas  may  better  predict 
student  performance,  a  new  assessment  will  have  to  be  developed  and  the  validation 
process  performed  once  again.  If  you  and  the  instructors  still  believe  the  areas 
assessed  impact  actual  performance,  you  should  look  at  additional  causes  for  the  lack 
of  validation. 

Questions  Used 

You  and  the  instructors  may  look  at  the  assessment  questions  used  to  measure  the 
level  of  prior  knowledge.  Remember,  the  idea  behind  the  questions  is  to  provide  a 
sample  of  individual  student  knowledge  that  indicates  how  much  that  student  knows 
about  some  particular  subject  or  knowledge  area.  Review  the  questions  to  be  sure  they 
cover  a  good  sampling  of  facts  or  topics  in  each  particular  knowledge  area. 

Review  the  section  in  this  guide  on  question  development  to  determine  if  the  questions 
follow  the  guidelines  described.  Make  sure  that  question  difficulty  and  question 
cognitive  levels  are  geared  toward  identifying  the  students  you  want.  In  evaluating  each 
question,  consider  the  following  general  points  which  are  described  in  further  detail  in 
Chapter  2  and  are  generally  applicable  to  all  question  formats. 

•  Ask  questions  involving  significant  or  major  content  and  avoid  trivial 
information. 

•  Make  sure  the  question  is  clear. 

•  Include  directions  that  clearly  state  the  basis  for  answering  the  question. 

•  Avoid  long,  complex  questions  while  still  ensuring  the  question  is  fully 
stated. 

•  Ensure  questions  are  stated  so  there  is  only  one  interpretation  of  the 
meaning  and  only  one  correct  response  for  each  question. 

•  Avoid  trick  questions. 

•  Avoid  negatively  stated  questions. 

•  Avoid  absolute  encompassing  words  such  as  “always”,  “all”,  or  “never”. 

•  Consider  the  use  of  diagrams,  pictures,  charts,  tables  and  figures  for 

application  of  principles  and  concepts. 
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Each  question  should  be  evaluated  to  ensure  it  is  asking  for  appropriate  knowledge  in 
an  appropriate  manner.  It  may  also  be  useful  to  analyze  student  responses  for  each 
question  to  aid  in  identifying  possible  problems  with  individual  questions.  For  example, 
if  most  students  answered  a  particular  question  incorrectly,  or  gave  similar  incorrect 
responses  to  a  question,  you  may  look  at  their  responses  to  determine  if  there  is  an 
issue  with  the  question  itself.  It  may  be  that  the  question  itself  or  the  question  directions 
were  not  clear  or  were  misleading. 

Although  beyond  the  scope  of  the  discussion  within  this  guide,  there  are  more  options 
for  evaluating  a  prior  knowledge  assessment  for  those  looking  for  a  further  analysis.  Of 
note,  a  test  item  analysis  may  also  be  conducted  on  either  the  assessment,  the  test 
from  which  a  criterion  score  was  obtained,  or  both.  A  test  item  analysis  provides 
statistics  on  overall  test  performance  and  individual  test  questions.  It  helps  in  identifying 
questions  that  might  be  poor  discriminators  of  student  performance  and  can  be  used  to 
improve  questions  for  future  test  administrations.  An  item  analysis  includes  two  main 
statistics  that  help  analyze  test  questions.  The  first  looks  at  question  difficulty  and 
shows  the  percentage  of  students  who  selected  the  correct  response.  The  second 
involves  item  effectiveness/discrimination  and  indicates  how  well  the  question  separates 
the  students  who  know  the  material  well  from  those  who  did  not.  For  a  further 
understanding  of  these  concepts  and  some  assistance  on  setting  up  the  calculations  in 
Excel  visit  the  following  websites: 

Penn  State,  Schreyer  Institute  for  Teaching  Excellence,  Improve  Multiple  Choice  Test 
Using  Item  Analysis  found  at:  http://www.schreyerinstitute.psu.edu/Tools/ltemAnalysis/ 
and  Selection  Matters  Blog  found  at:  http://selectionmatters.jmadiqan.net/?p=124 

Criterion  Used 

After  reviewing  the  assessment  for  subject  content  and  question  presentation  you  and 
the  instructors  may  also  look  outside  of  the  assessment  itself  for  other  potential  causes 
for  not  validating.  Remember  that  the  assessment  score  is  only  one  of  two  measures 
used  in  validation  with  the  other  being  the  criterion  that  measures  actual  performance. 
You  and  the  instructors  should  evaluate  the  criterion  used  in  validation  to  ensure  it 
measures  what  was  intended.  Chapter  3,  paragraph  3.3  discusses  the  development  of 
the  criterion  measure  used  for  validation.  In  that  discussion,  we  indicated  that  course 
performance  measurements  from  course  ratings,  scores,  or  rankings  need  to  be  those 
that  are  potentially  impacted  by  the  presence  of  prior  knowledge,  or  lack  thereof.  They 
should  not  however,  include  ratings  based  on  performance  not  relevant  or  potentially 
not  relevant  to  the  presence  of  the  prior  knowledge  areas  you  identified. 

Review  of  the  criterion  measure  may  reveal  inadvertent  inclusions  of  irrelevant 
measurements.  In  this  instance,  it  still  may  be  possible  to  use  the  criterion  by  removing 
any  irrelevant  measurements.  In  such  cases,  after  obtaining  new  criterion  scores  that 
measure  only  applicable  student  performance,  you  may  once  again  attempt  to  validate 
your  assessment  using  the  original  assessment  scores  and  the  new  criterion  scores. 
Another  potential  issue  with  the  criterion  may  be  that  student  scores  are  not  dispersed 
well  enough  to  make  a  correlation.  For  example,  if  nearly  all  student  criterion 
measurements  were  similar,  then  the  dispersion  between  the  scores  would  not  have  a 
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large  enough  spread  to  correlate  with  assessment  scores.  In  either  case  you  should 
check  to  ensure  the  criterion  is  measuring  what  you  intended. 

Timing  of  Criterion  Measurement 

Once  you  are  comfortable  that  your  criterion  is  measuring  what  you  intended,  you  may 
also  consider  the  timing  of  when  those  scores  were  obtained.  How  and  when  the 
criterion  measurements  were  taken  may  potentially  impact  your  validation  results.  This 
can  be  true  for  any  criterion  measurements,  but  is  especially  true  for  dichotomous 
measurements  such  as  Go/No-Go  scoring.  Consider  a  case  where  criterion  scores  are 
obtained  for  30  students  at  the  end  of  training.  Suppose  all  30  students  received  an 
overall  “Go”  for  the  course.  Comparing  prior  knowledge  assessment  scores  to  all  “Go” 
scores  tells  you  nothing  about  which  students  had  more  or  less  difficulty  with  the 
training  content.  Although  all  students  received  the  same  rating  at  the  end  of  training, 
some  may  have  struggled  much  more  than  others  to  obtain  that  rating.  In  this  case,  it 
would  be  better  to  obtain  criterion  scores  for  students  earlier  during  training  to  help 
distinguish  those  who  had  more  difficulty  with  the  content  from  those  who  found  it 
easier.  Review  the  figures  in  paragraph  3.3  of  this  guide  regarding  the  timing  of 
criterion  measures  to  see  if  they  were  taken  at  the  most  appropriate  time.  As  discussed 
in  this  section,  you  should  also  consider  alternatives  to  using  dichotomous 
measurements  that  could  better  distinguish  individual  student  performance. 

Recording  continuous  numerical  or  nondichotomous  criterion  measurements  late  in 
training  can  also  affect  validation  results.  Take,  for  example,  an  assessment  looking  to 
identify  students  who  could  benefit  from  additional  training  in  prior  knowledge  areas 
early  in  a  course.  The  more  exposure  students  have  to  other  students  and  training 
related  content,  the  more  knowledge  they  gain  even  if  not  specifically  taught  in  the 
course.  In  other  words,  the  more  association  students  have  with  information  on 
knowledge  areas  covered  in  the  assessment,  the  more  likely  they  are  to  use  that 
knowledge  in  actual  performance  related  criterion  measures.  In  an  overall  training 
sense,  this  is  not  a  problem  as  the  intent  is  for  all  students  to  perform  well  in  training. 
However,  the  intent  for  prior  knowledge  assessment  validation  is  to  determine  if  a 
relationship  of  prior  knowledge  to  later  performance  exists.  Criterion  scores  obtained 
late  in  training  may  obscure  the  fact  that  some  students  initially  struggled  with  training 
due  to  the  lack  of  prior  knowledge.  In  this  case,  it  would  be  beneficial  to  obtain  criterion 
measurements  at  some  earlier  point  in  training  rather  than  waiting  until  the  end  as 
shown  in  Figure  1,  Course  X,  in  Chapter  3. 

4.4  When  Do  I  Need  To  Revalidate  the  Assessment? 

Prior  knowledge  assessments  are  generally  developed  for  a  specific  block  of  instruction 
or  course.  Once  developed  and  validated,  the  prior  knowledge  assessment  should  be 
useful  for  subsequent  training  for  the  specific  course  for  which  it  was  designed. 
Generally,  the  assessment  will  remain  useful  when  administered  to  similar  groups  of 
students  attending  that  particular  training  course.  Remember  that  the  assessment  was 
validated  by  establishing  that  a  relationship  exists  between  students’  level  of  prior 
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knowledge  in  certain  knowledge  areas  and  their  performance  in  training.  However, 
changes  made  to  the  training  content  or  conditions  could  affect  that  relationship.  If 
changes  are  made  to  the  training  content  in  which  a  prior  knowledge  assessment  is 
used,  you  should  consider  whether  those  changes  have  altered  the  prior  knowledge 
areas  deemed  to  be  useful  for  that  training. 

Changes  that  alter  or  remove  the  basis  for  which  the  prior  knowledge  areas  were 
chosen  may  render  the  assessment  not  valid  for  use.  Any  changes  to  course  content 
should  be  evaluated  to  determine  whether  or  not  those  changes  effect  what  prior 
knowledge  was  sampled  in  the  assessment.  If  the  basis  for  which  those  prior 
knowledge  areas  was  removed  as  a  result  of  the  changes  then  new  knowledge  areas 
will  need  to  be  identified,  a  new  assessment  designed,  and  validation  procedures 
performed  once  again. 

If  you  and  the  instructors  determine  that  the  changes  to  course  content  had  little  to  no 
effect  on  the  prior  knowledge  areas  chosen,  then  the  assessment  can  continue  to  be 
used  for  making  tailored  training  decisions.  To  double  check  the  validity  of  the 
assessment  you  can  collect  criterion  measurements,  in  the  same  manner  as  done  in 
assessment  validation,  and  compute  a  new  correlation  coefficient.  Revalidation  of  the 
assessment  comes  from  obtaining  a  new  correlation  coefficient  of  0.3  or  above. 

4.5  What  Else  Can  the  Data  Tell  Me? 

Using  prior  knowledge  assessments  to  identify  students  for  tailored  training  can  help 
instructors  enhance  students’  learning  experience,  facilitate  transfer  of  knowledge,  and 
increase  student  performance  in  the  classroom.  Along  with  the  implementation  of 
tailored  training,  other  factors  can  also  affect  their  learning  experience.  Consider  how 
the  following  factors  may  affect  student  performance  in  the  classroom. 

•  Different  instructors  -  Instructors  have  their  own  unique  way  of  delivering 

content  during  training.  Given  two  identical  courses  with  identical  content,  two 
instructors  will  have  their  own  instructional  strategies  and  may  even  emphasize 
different  aspects  of  training  content.  Information  may  potentially  be  received 
differently  by  a  particular  student  depending  on  which  instructor  delivered  the 
training  and  how  it  was  delivered.  If  a  new  or  different  instructor  is  introduced 
into  the  classroom,  it  can  have  an  effect  on  student  performance  even  without 
the  benefit  of  tailored  training. 

•  Different  instructional  Strategies  -  Instructors  often  adjust  their  training 

strategies  as  they  become  more  familiar  with  instructing,  instructional 
techniques,  training  material,  and  course  content.  As  instructors  evolve,  they 
may  either  purposely  or  inadvertently  alter  their  delivery  or  delivery  techniques 
to  accommodate  how  students  respond  best.  Even  though  content  may  not 
have  changed,  these  changes  by  the  instructor  may  also  dramatically  affect 
student  performance. 
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•  New/Different  Equipment  or  Training  Aids  -  As  with  different  instructional 
strategies,  introducing  new  or  different  equipment  or  training  aids  can  alter  what 
students  learn  and  how  students  respond  to  training. 

Regardless  of  how  a  student’s  learning  experience  is  affected,  whether  from  tailored 
training  or  one  of  the  additional  factors  above,  the  intent  is  to  increase  student 
performance.  To  check  improvements  in  student  performance,  you  can  compare 
student  performance  results  from  validation  efforts  to  performance  results  after 
implementation  of  tailored  training.  In  order  to  accomplish  this,  you  should  measure 
your  criterion  in  a  class  that  received  tailored  training  in  the  same  manner  it  was 
measured  for  validation.  The  data  can  be  displayed  via  a  scatter  plot  using  methods 
similar  to  those  used  in  validation.  By  visually  comparing  the  scatter  plot  used  in  your 
original  validation  to  the  new  scatter  plot  generated,  you  should  expect  some  change  in 
performance  measurements.  Visualization  of  this  data  will  simply  help  you  track 
whether  tailoring  and  other  changes  improve  student  performance  and  should  be  used 
in  subsequent  training  in  which  the  prior  knowledge  assessment  is  used. 
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Appendix  B 

Editing  Input  Data  in  Excel 
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2.  “Double  click”  with  the  left  mouse  button  in  the  cell  where  you  entered  the 
correlation  function  and  you  will  see  the  function  in  the  cell  and  the  two  boxes 
appear  that  encompass  your  original  input. 
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3.  Take  the  cursor  and  point  to  “array  1”  in  the  function  and  click  the  left  mouse 
button.  Array  1  will  highlight  in  bold. 
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4.  As  before,  take  the  cursor  and  position  it  over  the  first  number  in  the  Assessment 
Score  column,  cell  “C2”,  click  the  left  mouse  button  and  while  holding  it  down 
drag  to  the  last  number  in  that  column,  to  include  the  new  entry,  and  release  the 
left  mouse  button. 
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5.  Take  the  cursor  and  point  to  “array  2”  in  the  function  and  click  the  left  mouse 
button.  Array  2  will  highlight  in  bold. 


Guide  Example.xlsx  -  Microsoft  Excel 


'm? 

Home  Insert  Page  Layout  Formulas  Data  Review 

View 

i  1  Page  Break  Preview 

is 

New  Window  3  JJ  j 

Ly  J 

I  j]  Custom  Views 

3  Arrange  All 

INorma'I 

Page 

Show/Hide 

Zoom  100%  Zoom  to 

5 

r 

Layout  '3  Ful1  Screen 

- 

Selection 

Freeze  Panes  *  3  lU  vVor 

Workbook  Views 

Zoom 

Window 

CONFIDENCE  ▼ 

*  f* 

- 

CORREL(C2:C33y  D2:D32) 

A 

B 

C 

D 

E 

F 

1 

Roster  # 

Student  Nam 

e  Assessment  Score 

Criterion  Score 

Correlation 

2 

2101 

Smith 

7 

} 

66 

i 

l 

77 

= 

CORRELrC2:C33.IBHUHJi 

3 

2102 

Jones 

| 

f 

67 

l 

) 

89 

CORREL(arrayl.  array2)  1 

4 

2103 

Davis 

} 

} 

77 

1 

87 

5 

2104 

Brown 

[ 

f 

66 

l 

i 

88 

6 

2105 

Austin 

J 

f 

72 

l 

77 

7 

2106 

Daniel 

[ 

I 

77 

l 

< 

45 

8 

2107 

Greene 

} 

} 

55 

1 

i 

66 

9 

2108 

Hawk 

[ 

( 

86 

1 

i 

92 

10 

2201 

Marshall 

} 

77 

i 

i 

92 

11 

2202 

Pate 

f 

77 

t 

l 

78 

12 

2203 

Martin 

f 

} 

89 

i 

90 

13 

2204 

Reed 

[ 

} 

78 

I 

i 

82 

14 

2205 

Starnes 

t 

} 

75 

l 

l 

83 

IS 

2206 

Thomas 

67 

i 

r 

] 

77 

16 

2207 

Reaves 

| 

66 

■ 

< 

77 

17 

2208 

Knight 

r 

67 

i 

i 

_ j _ 

89 

18 

2301 

Hunt 

[ 

77 

V 

1 

1 

88 

19 

2302 

Glover 

| 

1 

77 

1 

i 

_ i _ 

70 

20 

2303 

Hudson 

[ 

72 

V 

i 

( 

66 

21 

2304 

Grant 

r 

[ 

66 

l 

_ 

88 

22 

2305 

Barker 

| 

_ 1 _ 

65 

l 

1 

86 

23 

2306 

Miller 

r 

[ 

j 

76 

i 

t 

_ J _ 

87 

24 

2307 

Abell 

[ 

f 

78 

1 

2 

J 

62 

25 

2308 

Cook 

r 

f 

\ 

84 

1 

2 

i 

89 

26 

2401 

Dent 

1 

75 

■ 

f 

i 

88 

27 

2402 

Franks 

- 1 - 

f 

\ 

79 

i 

i 

j 

82 

28 

2403 

Henson 

_ 1 _ 

67 

i 

i 

i 

79 

29 

2404 

Jameson 

r 

f 

| 

69 

i 

f 

_ j _ 

88 

30 

2405 

Kraft 

| _ 

65 

r 

l 

1 

80 

31 

2406 

Walker 

} 

) 

81 

1 

r 

j 

93 

32 

2407 

Tanner 

[ 

t 

72 

I 

i 

L 

84 

33 

2408 

Williams 

- 1 - 

f 

i. 

76 

r 

90 

34 

GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING 


6.  Similarly,  take  the  cursor  and  position  it  over  the  first  number  in  the  Criterion 
Score  column,  cell  “D2”,  click  the  left  mouse  button  and  while  holding  it  down 
drag  to  the  last  number  in  that  column,  to  include  the  new  entry,  and  release  the 
left  mouse  button. 


□  J] 

J  Page  Break  Preview 

\j  pr-  j  New  Window 

IjJ  Custom  Views 

^  Arrange  All 

IKlormal  Page 

Show/Hide 

Zoom 

100%  Zoom  to 

1  Layout 

3)  Full  Screen 

Selection  P  Freeze  Panes  ’ 

□  *13  wot 

Workbook  Views 

Zoom 

Window 

0537 

Home 


Insert  Page  Layout  Formulas  Data  Review  View 


CONFIDENCE 


X  «/  Ji  =CORREL(C2:C33,D2:D33) 


A 

BCD  Mk.  E 

F 

1 

Roster  # 

Student  Name  Assessment  Score  Criterion  Score  Correlation 

2 

2101 

Smith 

66  T  77  ]=CORREL(C2:C33,D2:D33) 

3 

2102 

Jones 

67  |  89  j  CORREL(arrayX  array2) 

4 

2103 

Davis 

//  JO /  i 

5 

2104 

Brown 

66  j  88  j 

6 

2105 

Austin 

72  1  77  i 

7 

2106 

Daniel 

11  j  45  j 

8 

2107 

Greene 

55  !  66  I 

9 

2108 

Hawk 

86  j  92  j 

10 

2201 

Marshall 

77  |  92  1 

11 

2202 

Pate 

77  f  78  | 

12 

2203 

Martin 

89  !  90  i 

13 

2204 

Reed 

78  j  82  j 

14 

2205 

Starnes 

75  j  83  I 

15 

2206 

Thomas 

67  J  77  j 

16 

2207 

Reaves 

66  f  77  { 

17 

2208 

Knight 

67  j  89  j 

18 

2301 

Hunt 

77  I  88  I 

19 

2302 

Glover 

77  !  70  I 

20 

2303 

Hudson 

72  I  66  1 

21 

2304 

Grant 

66  j  88  j 

22 

2305 

Barker 

65  |  86  I 

23 

2306 

Miller 

76  j  87  j 

24 

2307 

Abell 

78  I  62  1 

25 

2308 

Cook 

84  j  89  j 

26 

2401 

Dent 

75  I  88  j 

27 

2402 

Franks 

79  j  82  j 

28 

2403 

Henson 

67  !  79  I 

29 

2404 

Jameson 

69  j  88  j 

30 

2405 

Kraft 

65  !  80  j 

31 

2406 

Walker 

81  j  93  j 

32 

2407 

Tanner 

72  j  84  I 

33 

2408 

Williams 

76  1 ?o 

34 

GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  B-7 


7.  You  will  notice  that  as  you  increase  the  size  of  the  boxes  the  corresponding  cell 
positions  automatically  change  in  your  formula.  Once  both  boxes  have  been 
resized,  hit  the  “Enter”  key  and  your  new  value  including  the  new  entry  will  be 
calculated. 
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Changing  Input  Values 

1 .  If  you  found  that  you  made  an  input  error,  you  may  change  any  of  the  data  you 
entered,  hit  the  “Enter”  key,  and  it  will  automatically  update  the  correlation 
coefficient.  The  examples  below  show  continuous  numerical  measurement  data. 
The  procedures  are  the  same  for  dichotomous  data. 
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2.  The  same  process  is  true  for  dichotomous  criterion  score  data.  Simply  change 
any  of  the  data  you  entered,  hit  the  “Enter”  key,  and  the  correlation  coefficient  will 
be  updated  automatically. 
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Deleting  Data 

1 .  If  you  need  to  delete  an  entire  entry,  click  the  right  mouse  button  on  the  number 
of  the  entry  to  be  deleted  which  will  highlight  all  of  the  cells  pertaining  to  that 
entry.  This  will  also  bring  up  a  menu.  The  examples  below  show  continuous 
numerical  measurement  data.  The  procedures  are  the  same  for  dichotomous 
data. 
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2.  Inside  the  menu  click  on  “Delete”  and  the  entire  row/entry  will  be  deleted. 

CAUTION:  Deleting  line  2,  which  includes  the  correlation  formula,  will  also 
delete  the  formula. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  B-1 2 


3.  Notice  as  the  entry  is  deleted  the  correlation  coefficient  is  automatically  updated. 
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Sorting  Data 

1 .  To  sort  assessment  scores  for  ease  in  visualization  of  the  data,  left  click  on  the 
first  cell  containing  the  label  for  the  first  column  (Roster  #),  drag  to  the  last  cell  in 
the  last  column  and  release  the  button.  This  will  highlight  all  of  the  data  including 
the  column  headings.  All  columns  must  be  highlighted  in  order  to  maintain 
the  association  of  student  roster  numbers,  names,  and  scores.  Without 
selecting  all  columns  only  a  portion  of  the  data  will  be  sorted  and  scores 
will  become  misaligned  with  student  names  and/or  roster  numbers. 
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2.  Left  click  on  the  “Data”  tab  at  the  top  of  the  screen  and  then  left  click  on  “Sort”  in 
the  tool  bar.  This  will  bring  up  a  “Sort”  menu. 
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3.  In  the  menu,  click  on  the  down  arrow  next  to  “Sort  by”  revealing  the  options. 
Move  your  pointer  to  “Assessment  Score”  and  left  click  on  that  option.  “Sort  On” 
should  remain  on  “values”  and  “Order”  should  remain  on  “Smallest  to  Largest”. 
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4.  Click  on  “OK”  at  the  bottom  of  the  menu  screen  and  your  assessment  scores  will 
be  sorted  along  with  the  associated  roster  number  and  student  name. 
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Appendix  C 


Excel  Correlation  Coefficient 
Sample  Exercises 


The  sample  exercises  located  in  this  appendix  are  provided  in  order  for  you  to  gain 
experience  and  confidence  in  using  Excel  to  determine  correlation  coefficients  and 
produce  scatter  plots.  Using  the  Excel  software  located  on  your  computer,  input  the 
data  provided  in  the  examples  to  produce  the  requested  output.  If  you  need  assistance, 
refer  to  the  appropriate  section  in  this  guide  for  a  detailed  explanation. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING 


C-1 


Exercise  #1  -  Obtain  a  Correlation  Coefficient 


You  have  completed  instruction  on  a  block  of  training  in  which  you  plan  to 
implement  tailored  training.  Prior  to  beginning  this  instruction  you  designed  a 
prior  knowledge  assessment  to  be  used  for  validation  and  administered  it  to 
the  class.  You  have  just  finished  scoring  the  assessment  and  have  compiled 
student  performance  scores  used  for  your  criterion.  Using  the  information 
found  in  this  guide,  calculate  a  correlation  value  from  your  data  below. 
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Once  you  have  obtained  a  correlation  coefficient  turn  to  the  next  page. 
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Exercise  #1  (continued) 


You  should  have  obtained  a  value  of  .21 .  If  you  did  not,  check  your  input  and 
try  again.  You  should  only  use  data  for  students  in  which  you  have  both 
scores.  Missing  scores  may  be  due  to  recycling  or  various  other  reasons  but 
partial  scores  may  not  be  included.  Scores  circled  in  red  may  not  be  used  in 
the  calculation. 
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Once  you  have  successfully  calculated  a  value  of  .21  then  continue  on  to  the  next  page. 
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Exercise  #2  -  Create  a  Scatter  Plot 


Your  Excel  spreadsheet  should  look  similar  to  the  one  below. 

You  now  decide  to  use  a  scatter  plot  to  help  identify  any  outliers.  Create  a 
scatter  plot  using  your  input  data. 
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Once  you  have  successfully  created  a  scatter  plot  continue  on  to  the  next  page. 
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Exercise  #3  -  Correct  Input  Data 


Your  Excel  spreadsheet  should  look  similar  to  the  one  below.  If  not  review 
the  instructions  on  creating  scatter  plots  and  try  again. 

Looking  at  the  scatter  plot  you  observe  that  the  data  point  for  student  2151 
appears  to  fall  outside  the  cluster  of  others.  Double  checking  the  data  you 
find  that  you  made  a  data  entry  error  and  his  assessment  score  should  have 
been  41  rather  than  14.  Correct  the  score  and  obtain  a  new  correlation  value 
and  scatter  plot. 
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Once  you  have  successfully  changed  the  data  and  obtained  a  new  correlation 
coefficient  continue  on  to  the  next  page. 
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Exercise  #3  (continued) 


You  should  have  obtained  a  value  of  .32  and  seen  the  associated  data  point 
move  to  a  different  location.  If  you  did  not,  check  your  input  and  try  again. 

Your  Excel  spreadsheet  should  look  similar  to  the  one  below. 

To  change  the  data  simply  click  in  cell  B18  and  replace  14  with  41  and  hit 
“enter”.  This  will  automatically  update  the  correlation  value  and  scatter  plot. 
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Once  you  have  obtained  a  value  of  .32  and  successfully  created  the  scatter  plot  as 
pictured  above,  continue  on  to  the  next  page. 
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Exercise  #4  -  Input  New  Data 


As  you  are  checking  the  data  you  realize  you  had  not  recorded  any  data  for 
student  2121.  Student  2121  had  an  assessment  score  of  32  and  a  criterion 
score  of  42.  Include  the  new  values  into  your  input  data  and  recalculate  the 
correlation  value  and  obtain  a  new  scatter  plot. 
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Once  you  have  added  the  student  data,  obtained  a  new  correlation  coefficient  and 
associated  scatter  plot,  continue  on  to  the  next  page. 


GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING  C-7 


Exercise  #4  (continued) 


You  should  have  obtained  a  value  of  .35.  If  you  did  not,  check  your  input  and 
try  again.  Your  Excel  spreadsheet  should  look  similar  to  the  one  below. 
Remember,  you  have  to  update  the  cell  information  for  both  the  correlation 
and  the  scatter  plot. 
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Continue  on  to  the  next  page  for  further  help  in  obtaining  the  correct  value  and  adding 
the  data  point  to  the  scatter  plot. 
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Exercise  #4  (continued) 


To  insert  the  new  data,  add  the  data  to  the  bottom  of  the  list.  Double  click  on 
cell  D2,  click  on  “array  1  ”  in  the  formula,  highlight  all  of  the  assessment 
scores,  click  on  “array  2”,  highlight  all  of  the  criterion  scores  and  then  hit  the 
enter  key  on  your  keyboard  to  encompass  the  new  data.  This  will 
automatically  update  the  correlation  value  but  not  the  scatter  plot. 


Turn  to  the  next  page  for  assistance  on  how  to  update  the  scatter  plot. 
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Exercise  #4  (continued) 


There  are  a  number  of  ways  to  update  the  scatter  plot.  The  simplest  way  is  to 
click  the  right  mouse  button  somewhere  inside  the  scatter  plot  which  will  bring 
up  a  menu  and  then  click  the  left  mouse  button  on  “Select  data”. 


Once  you  have  clicked  on  “Select  data”  turn  to  the  next  page. 
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Exercise  #4  (continued) 


Once  you  have  clicked  on  “Select  data”  you  will  see  your  input  data  for  the 
scatter  plot  highlighted  in  a  flashing  dotted  line  along  with  a  data  source  box. 
Your  input  data  will  not  include  your  new  entry.  To  update  your  data  simply 
click  your  left  mouse  button  on  the  first  assessment  score  entry  and  then, 
without  releasing  the  button,  drag  to  the  last  criterion  score  entry.  Now 
release  the  left  mouse  button.  This  will  draw  a  new  box  around  all  of  the  data 
and  update  the  data  in  the  data  source  box.  Once  you  have  drawn  the  new  box 
around  all  of  the  data,  click  “OK”  at  the  bottom  of  the  data  source  box. 


Once  you  have  updated  all  data  correctly  turn  to  the  next  page. 
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Exercise  #4  (continued) 


Once  you  click  “OK”  at  the  bottom  of  the  data  source  box  the  updated  scatter 
plot  will  appear  with  the  new  data  point  displayed. 


Once  you  have  updated  your  scatter  plot,  turn  to  the  next  page. 
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C-12 


Exercise  #5  -  Delete  Data 


Just  as  you  think  you  have  finished  you  remember  that  student  2144  became 
sick  on  the  day  the  assessment  was  administered  and  did  not  finish  the 
assessment.  You  decide  to  exclude  that  students  data  from  the  analysis. 
Delete  all  data  for  student  2144  and  update  both  the  correlation  value  and  the 
scatter  plot. 


Once  you  have  deleted  student  2144  and  updated  all  data  correctly  turn  to  the  next 
page. 
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Exercise  #5  (continued) 


You  should  have  obtained  a  correlation  value  of  .46  and  seen  the  associated 
data  point  disappear.  If  you  did  not,  check  your  input  and  then  turn  to  the  next 
page  for  some  helpful  hints. 
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Continue  on  to  the  next  page  for  further  help  in  obtaining  the  correct  value  and 
removing  the  data  point  from  the  scatter  plot. 
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Exercise  #5  (continued) 


To  delete  the  data  simply  highlight  the  entire  row  by  clicking  on  the  row 
number  with  the  right  mouse  button  which  will  also  bring  up  a  menu.  Once  the 
row  is  highlighted  mouse  over  the  “delete”  option  and  click  the  left  mouse 
button.  When  the  second  menu  appears  click  the  “Shift  cells  up”  button  and 
then  click  on  “OK”.  This  will  delete  all  data  associated  with  that  row  and  the 
correlation  value  and  scatter  plot  will  be  updated  automatically. 
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Exercise  #6  -  Obtain  a  Correlation  Coefficient  with 

Dichotomous  Data 


Using  the  information  found  in  this  guide  for  obtaining  a  correlation  coefficient 
from  data  containing  a  dichotomous  measurement,  calculate  a  value  from  the 
data  below. 

Assessment  scores  Criterion  Scores 


Roster  # 

Score 

Roster  # 

Score 

2114 

44 

2114 

1 

2115 

42 

2115 

0 

2117 

37 

2117 

1 

2124 

35 

2124 

1 

2125 

33 

2125 

0 

2127 

39 

2126 

0 

2128 

47 

2127 

0 

2129 

41 

2128 

1 

2132 

38 

2132 

1 

2135 

29 

2135 

0 

2137 

34 

2137 

0 

2139 

35 

2139 

1 

2144 

28 

2142 

0 

2145 

46 

2144 

0 

2146 

42 

2145 

1 

2147 

45 

2146 

0 

2148 

47 

2148 

1 

2150 

43 

2150 

0 

2151 

14 

2151 

0 

2152 

33 

2152 

0 

2154 

36 

2157 

1 

2157 

40 

2158 

1 

2158 

44 

2160 

1 

2160 

45 

2161 

0 

2161 

39 

2162 

1 

2162 

33 

2164 

1 

2164 

29 

2165 

0 

2165 

27 

2167 

1 

2167 

35 

2169 

1 

2169 

46 

2170 

0 

2170 

42 

2171 

1 

2171 

41 

2172 

1 

2172 

40 

2173 

0 

2173 

29 

2174 

0 

2174 

30 

Once  you  have  obtained  a  value,  continue  on  to  the  next  page. 
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Exercise  #6  -  Obtain  a  Correlation  Coefficient  with  Dichotomous  Data 


You  should  have  obtained  a  value  of  .39.  If  you  did  not,  check  your  input  and 
try  again.  Your  Excel  spreadsheet  should  look  similar  to  the  one  below. 
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Assessment  Score 

Criterion  Score 

Correlation 

2 

2114 

44 

1 

0.39 

3 

2115 

42 

0 

4 

2117 

37 

i 

* 

5 

2124 

35 

i 

6 

2125 

33 

0 
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2127 

39 

0 
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2128 

47 

1 

9 

2132 

38 

1 

10 

2135 

29 

0 

11 

2137 

34 

0 

12 

2139 

35 

1 

13 

2144 

28 

0 

14 

2145 

46 

1 

15 

2146 

42 

0 

16 

2148 

47 

1 

17 

2150 

43 

0 

18 

2151 

41 

0 

19 

2152 

33 

0 

20 

2157 

40 

1 

21 

2158 

44 

1 

22 

2160 

45 

1 

23 

2161 

39 

0 

24 

2162 

33 

1 

25 

2164 

29 

1 

26 

2165 

27 

0 

27 

2167 

35 

1 

28 

2169 

46 

1 

29 

2170 

42 

0 

30 

2171 

41 

1 

31 

2172 

40 

1 

32 

2173 

29 

0 

33 

2174 

30 

0 

Once  you  have  reviewed  your  data  with  the  spreadsheet  above,  continue  on  to  the  next 
page  for  some  helpful  hints  on  common  errors. 
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Exercise  Helps  -  Helpful  Tips  for  Common  Errors 


If  you  are  having  trouble  with  obtaining  the  correct  answers  for  any  of  the 
exercises,  review  the  list  of  tips  below  for  further  assistance. 

1.  Make  sure  you  have  only  entered  data  for  students  with  both  an 

assessment  and  a  criterion  score. 

2.  Ensure  that  the  proper  scores  are  recorded  for  the  proper  student 

name/roster  number. 

3.  Ensure  that  all  calculations  begin  with  an  equal  (=)  sign. 

4.  Ensure  that  functions  that  include  an  array,  such  as  “=SUM(D2:D33)”, 

include  all  of  the  values  it  should  have  without  including  the  heading 
name  for  the  array.  You  can  check  the  cell  identifiers,  such  as  “D2” 
and  “D33”,  within  the  function  to  ensure  all  are  correctly  entered. 

5.  Ensure  all  formulas  include  the  correct  cell  identifiers  as  well  as  the 

correct  mathematical  symbols. 

6.  Ensure  all  parentheses  are  placed  in  the  correct  position  inside 

formulas. 

7.  Ensure  that  if  you  have  added  data  that  you  update  the  corresponding 

formulas  to  include  the  new  data. 

8.  If  using  scatter  plots,  remember  that  scatter  plot  data  must  be  updated 

separately  when  changing  original  input  data. 

9.  If  any  sorting  of  input  data  has  been  accomplished,  ensure  all  data 

pertaining  to  a  particular  individual  moves  with  that  entry. 
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Appendix  D 


Example  Questions  Used  in  Actual 
Prior  Knowledge  Assessments 


The  following  questions  were  used  in  a  variety  of  different  prior  knowledge  assessments 
and  are  provided  as  further  examples  of  the  points  reviewed  in  this  guide. 


From  marksmanship  training: 

Directions:  Circle  the  answer  you  select. 

1 .  A  bullet  flying  through  the  air  is  acted  upon  primarily  by  two  forces  which  change  the 
direction  and  velocity  of  its  motion.  These  two  forces  are? 

a.  Temperature  and  Humidity 

b.  Elevation  and  Friction 

c.  Bullet  Weight  and  Caliber 

d.  Gravity  and  Air  Resistance 

e.  I  don’t  know 

2.  The  path  of  flight  that  the  bullet  will  take  when  it  is  fired  from  the  rifle  is  known  as 
what? 

a.  Max  ordinance 

b.  Trajectory 

c.  Terminal  ballistics 

d.  Physics 

e.  I  don’t  know 

3.  What  happens  when  a  bullet  leaves  the  bore  of  the  rifle  in  which  the  barrel  is 
horizontal  to  the  ground  and  the  line  of  sight  is  parallel  to  the  line  of  bore 

a.  It  will  fly  straight  until  it  hits  the  target 

b.  It  will  go  up  due  to  its  aerodynamic  properties 

c.  It  will  immediately  begin  to  fall  to  the  earth 

d.  It  depends  on  the  Ballistic  Coefficient 

e.  I  don’t  know 
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4.  How  many  times  will  the  bullet  cross  your  line  of  sight  before  it  hits  the  target  when 
engaging  a  350  yard  target  with  a  300  yard  Battle  Sight  Zero  while  aiming  center 
mass? 

a.  Once 

b.  Twice 

c.  Three  times 

d.  It  won’t  cross  at  any  time 

e.  I  don’t  know 

5.  If 


a. 

b. 

c. 

d. 

e. 

f. 
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From  land  navigation  training: 

1 .  Select  the  type  of  major  terrain  feature  identified  by  the  boxes. 


2.  An  eight  digit  coordinate  locates  a  point  on  the  ground  to  within _ ? 

a.  100  meters 

b.  1  meter 

c.  1 00  feet 

d.  10  meters 

3.  Military  topographic  maps  use  colors  and  symbols  to  depict  various  types  of  ground 
cover,  forest  types,  and  cultivation.  Match  the  symbol  with  the  type  of  plant  life  or 
vegetation  that  it  depicts.  Enter  the  letter  for  the  symbol  in  the  appropriate  blank. 

(NOTE:  Not  all  letters  may  match  descriptions;  if  no  match  is  found  place  an  “X”  in 
the  answer  space.) 
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_ Brush  and  scrub  _ Orchards  _ Rice  paddies  _ Open  fields 

4.  The  contour  interval  of  1 0  feet  is  identified  in  the  legend  of  for  this  map.  Which 
labeled  arrow  contains  the  steepest  climb?  (Circle  the  best  answer.) 
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5.  From  your  location  at  the  center  of  the  triangle  in  the  dashed  orange  box,  what  is  the 
direction  of  movement  to  the  road  junction  in  the  black  box?  (Circle  the  correct 
answer.) 


6.  You  remain  at  the  same  location  as  in  the  previous  question.  What  is  the 

approximate  grid  azimuth  to  the  road  junction  in  the  black  box  (Circle  the  correct 
answer) 

315°  225°  25°  157°  190°  362°  265° 
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From  engineer  operations  and  planning  training: 

The  example  below  depicts  varied  obstacle  groups  along  enemy  avenues  of 
approach  through  a  company  position.  Normally,  a  company-team  will  have  the 
mission  to  cover  only  one  or  two  obstacle  groups  in  the  defense.  Answer  questions 
18-19  referring  to  the  sketch  below: 


1 .  Match  the  Obstacle  Groups  with  the  desired  effect  that  the  commander  desires 
along  each  enemy  avenue  of  approach.  (Enter  the  letter  for  the  obstacle  effect 
beside  the  Obstacle  Group.  Obstacle  effects  may  be  used  more  than  once  or  not  at 
all.) 


A.  Disrupt  Effect 

Obstacle  Group  1 

B.  Obstruct  Effect 

Obstacle  Group  2 

C.  Turn  Effect 

Obstacle  Group  3 

D.  Block  Effect 

E.  Fix  Effect 
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2.  Along  which  Enemy  AA  would  you  expect  to  find  the  greatest  concentration  of 
planned  massed  direct  and  indirect  fires  integrated  with  the  obstacles?  (Select  one 
answer.) 


A.  Avenue  of  Approach  1 

B.  Avenue  of  Approach  2 

C.  Avenue  of  Approach  3 

D.  Planned  direct  and  indirect  fires  would  be  equally  distributed  across  all  AAs. 


3.  Identify  factors  that  you,  as  the  TF  Engineer,  should  consider  when  developing  the 
work  plan  and  obstacle  execution  matrix  for  the  available  engineers  and  excavation 
resources?  (Circle  all  that  apply;  one  or  more  responses  are  correct.) 

A.  Time  available  to  prepare  the  defense 

B.  Construction  materials  or  munitions  required  and  available 

C.  Priorities  established  by  the  maneuver  commander 

D.  Fuel  consumption  rates  for  all  excavation  systems,  troop  carriers,  and  prime 
movers 

E.  Excavation  or  emplacement  systems  available  and  their  projected  operational 
readiness  rate(s) 

F.  Travel  time  between  primary  work  sites 
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From  electrical  maintenance  training: 

1. 


A  B 


This  circuit  has  four  ’’test  points”  labeled  with  the  letters  A,  B,  C,  and  D.  Assuming  the 
circuit  is  functioning  (light  bulb  is  energized),  determine  whether  or  not  there  will  be 
substantial  voltage  between  the  following  sets  of  points: 


Between  A  and 

B: 

Voltage 

No  Voltage 

Don’t  Know 

Between  B  and 

C: 

Voltage 

No  Voltage 

Don’t  Know 

Between  C  and 

D: 

Voltage 

No  Voltage 

Don’t  Know 

Between  D  and 

A: 

Voltage 

No  Voltage 

Don’t  Know 

Between  A  and 

C: 

Voltage 

No  Voltage 

Don’t  Know 

Between  D  and 

B: 

Voltage 

No  Voltage 

Don’t  Know 

GUIDE  FOR  DEVELOPING  AND  USING  PRIOR  KNOWLEDGE  ASSESSMENTS  TO  TAILOR  TRAINING 


D-8 


2.  Examine  this  schematic  diagram: 


Switch 


Light  bulb 


Battery 


Now,  without  moving  the  following  components,  show  how  they  may  be  connected 
together  with  wires  to  form  the  same  circuit  depicted  in  the  schematic  diagram  above. 

Please  draw  lines  indicating  the  wires. 


Light  bulb 
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3.  Identify  the  “circuit  symbol”  with  its  corresponding  component. 


B. 


C. 


D. 


E. 


F _ rYYY\ 


H. 


I. 


Earth  (Ground) 

Motor 

Transistor  PNP 

Diode 

Lamp  (Indicator) 

Ammeter 

Voltmeter 

Ohmmeter 

Coil;  Solenoid 


Don’t  Know 


Don’t  Know 


Don’t  Know 


Don’t  Know 


Don’t  Know 

Don’t  Know 

Don’t  Know 


Don’t  Know 


Don’t  Know 
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